Light displays in a medical device

ABSTRACT

An example medical device may include a first component including an interface, a light feature surrounding at least part of the interface, and a controller coupled to the light feature and including a memory in which are stored instructions for the controller causing a first illumination state of the light a feature corresponding to a first state of the interface, and the controller causing a second illumination state of the Sight feature corresponding to a second state of the interface.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a U.S. National Stage Filing under 35 U.S.C. 371from International Application No. PCT/US2018/043747, filed on Jul. 25,2018, and published as WO 2019/023378 A1 on Jan. 31, 2019, which claimsthe benefit of priority to U.S. Provisional Patent Application No.62/537,884, filed on Jul. 27, 2017, each of which is incorporated byreference herein in its entirety.

This application is also related to the following international patentapplications filed with the U.S. receiving office on even date herewith:international application no. PCT/US2018/043757, entitled INTEGRALDISPLAY AND DISPLAY OF CONTENT; international application no.PCT/US2018/043764, entitled DISPLAY BETWEEN ARM LINKS IN A MEDICALDEVICE; international application no. PCT/us2018/043772, entitledMEDICAL DEVICE HANDLE; and international application no.PCT/US2018/043776, entitled MEDICAL DEVICE WITH ERGONOMIC FEATURES.

TECHNICAL FIELD

This document relates generally to medical devices, and moreparticularly, to user interface or ergonomic features in a medicaldevice such as a teleoperated surgical system.

BACKGROUND

A teleoperated surgical system that operates at least in part undercomputer control includes one or more teleoperated manipulator arms thateach include two or more interconnected links. The system may interfacewith other systems or devices, which may provide power or controlsignals. A surgical instrument may be mounted to an arm and controlledusing a user control system.

SUMMARY

An example (“Example 1”) of subject matter (e.g., a medical device) mayinclude a first component including an interface, a light featuresurrounding at least part of the interface, and a controller coupled tothe light feature and including a memory in which are storedinstructions for the controller causing a first illumination state ofthe light feature corresponding to a first state of the interface, andthe controller causing a second illumination state of the light featurecorresponding to a second state of the interface.

In Example 2, the subject matter of Example 1 may be configured suchthat the first state includes a first displacement value and the secondstate includes a second displacement value, and an aspect of the firstillumination state of the light feature corresponds to the firstdisplacement value, and an aspect of the second illumination state ofthe light feature corresponds to the second displacement value.

In Example 3, the subject matter of Example 2 may be configured suchthat the first illumination state includes a first illuminated length ofthe light feature corresponding to the first displacement value at theinterface, and the second illumination state includes a secondilluminated length of the light feature corresponding to a second jointdisplacement value at the interface.

In Example 4, the subject matter of Example 1 may be configured suchthat an aspect of the first illumination state of the light featurecorresponds to a first rotational state of the interface, and an aspectof the second illumination state corresponds to a second rotationalstate or a first translational state of the interface.

In Example 5, the subject matter of Example 4 may be configured suchthat the light feature indicates position relative to a range of motionlimit.

In Example 6, the subject matter of Example 1 may be configured suchthat a first aspect of the light feature corresponds to a rotationalstate and a second aspect of the light feature corresponds to atranslational state.

In Example 7, the medical device of Example 6 may be configured suchthat a length of the light feature in a transverse direction correspondsa translational displacement value, and a length of the light featureextending around the interface corresponds to a rotational displacementvalue.

In Example 8, the subject matter of any one or any combination ofExamples 1-7 may be configured such that the light feature includesmovable features and a first rate of movement of the movable features inthe first illumination state corresponds to a first velocity of movementof a movable portion of the medical device, and a second rate ofmovement of the movable features in the second illumination statecorresponds to a second velocity of movement of the movable portion ofthe medical device.

In Example 9, the subject matter of Example 1-8 may be configured suchthat the light feature indicates a kinematic pose of the medical device.

In Example 10, the subject matter of any one or any combination ofExamples 1-9 may be configured such that the light feature indicates asource of control commands for the medical device.

An example (“Example 11”) of subject matter (e.g., a teleoperatedsurgical system) may include a first link, a second link coupled to thefirst link at an interface, a light feature surrounding at least part ofthe interface and having a variable illumination state, and a controllercoupled to the light feature and including a memory in which are storedinstructions for controlling the illumination state of the light featurebased at least in part on a movement or configuration of the second linkwith respect to the first link.

In Example 12, the teleoperated surgical system of Example 11 may beconfigured such that the controller includes stored instructions tocontrol the illumination state of the light feature based on an angularrelationship of the second link with respect to the first link.

In Example 13, the teleoperated surgical system of Example 11 or 12 maybe configured such that the controller includes stored instructions tocontrol the illumination state of the light feature based on a kinematicpose of the first link and the second link.

In Example 14, the teleoperated surgical system of any one or anycombination Examples 11-13 may be configured such that the controllerincludes stored instructions to control the illumination state of thelight feature based on a range of motion.

In Example 15, the teleoperated surgical system of Examples 11-13 may beconfigured such that controller includes instructions for the controllercausing a first illumination state of the light feature corresponding toa first state of the interface, and the controller causing a secondillumination state of the light feature corresponding to a second stateof the interface.

In Example 16, the subject matter of Example 15 may be configured suchthat the first state includes a first displacement value and the secondstate includes a second displacement value, and an aspect of the firstillumination state of the light feature corresponds to the firstdisplacement value, and an aspect of the second illumination state ofthe light feature corresponds to the second displacement value.

An example (“Example 17) of subject matter (e.g., a method ofcontrolling a medical device having an interface) may include presentinga first illumination state of the light feature corresponding to a firststate of the interface, and presenting a first illumination state of thelight feature corresponding to a second state of the interface.

In Example 18, the method of Example 17 may be implemented or configuredsuch that the first illumination state corresponds to a first jointdisplacement value and the second illumination state corresponds to asecond joint displacement value.

In Example 19, the method of Examples 17 or 18 may be implemented ofconfigured such that the first illumination state corresponds to a firstjoint angle and the second illumination state corresponds to a secondjoint angle.

In Example 20, the method of Examples 17 or 18 may be configured orimplemented such that presenting the first illumination state includespresenting a first animated state in which an apparent movement of thelight feature corresponds to a first movement at the interface, andpresenting a second illumination state includes presenting a secondanimated state in which an apparent movement of the light featurecorresponds to a second movement at the interface.

Each of these non-limiting examples can stand on its own, or can becombined in various permutations or combinations with one or more of theother examples.

This Summary is intended to provide an overview of subject matter of thepresent patent application. It is not intended to provide an exclusiveor exhaustive explanation of the invention. The detailed description isincluded to provide further information about the present patentapplication.

BRIEF DESCRIPTION OF THE DRAWINGS

In the drawings, which are not necessarily drawn to scale, like numeralsmay describe similar components in different views. Like numerals havingdifferent letter suffixes may represent different instances of similarcomponents. The drawings illustrate generally, by way of example, butnot by way of limitation, various embodiments discussed in the presentdocument.

FIG. 1A is a plan view illustration of an example medical system thatmay include a user control system, an auxiliary system, and amanipulating system.

FIG. 1B is an illustration of an example manipulating system.

FIG. 1C is an illustration of an example user control system.

FIG. 1D is an illustration of an example auxiliary system.

FIG. 1E is an illustration of an example instrument.

FIG. 2A is a perspective illustration of an example manipulating system.

FIG. 2B is a side view illustration of the example manipulating systemshown in FIG. 2A.

FIG. 2C is an illustration of a manipulating system positioned over asurgical table.

FIG. 2D is an illustration of a manipulating system being used in asurgical procedure.

FIG. 2E is an illustration of an arm portion of a manipulating systemextending across a surgical table.

FIG. 2F is an illustration of two manipulating systems positionedside-by-side over a surgical table.

FIG. 2G is an illustration of a manipulating system in an “IV stand”kinematic pose.

FIG. 2H is an illustration of a manipulating system in a “ready todrape” kinematic pose.

FIG. 2I is an illustration of a manipulating system in a “ready to dock”kinematic pose.

FIG. 2J is an illustration in a “ready to transport” kinematic pose.

FIG. 3A is an illustration of an integrated display on an arm portion ofa manipulating system.

FIG. 3B is an illustration of an integrated display on an arm portion ofa manipulating system.

FIG. 3C is an illustration of an integrated display in an arm portion ofa manipulation system in a ready to drape kinematic pose.

FIG. 4A is an illustration of a display on an arm portion of amanipulating system.

FIG. 4B is an illustration of a display on an arm portion of amanipulating system and buttons on the arm.

FIG. 5A is an illustration of a handle on an arm of a manipulatingsystem.

FIG. 5B is an illustration of a manipulating system with a handlepresented to a user.

FIG. 6A is an illustration of a retractable handle on an arm of amanipulating system.

FIG. 6B is a side perspective view of the handle shown in FIG. 6A.

FIG. 7A is an illustration of a handle on a display.

FIG. 7B is a perspective illustration of the handle shown in FIG. 7A.

FIG. 7C is a top view of the handle shown in FIG. 7A and FIG. 7B.

FIG. 8 is an illustration of a helm.

FIG. 9A is an illustration of another example manipulating system.

FIG. 9B is an illustration of an example manipulating system and asurgical table.

FIG. 9C is an illustration of two manipulating systems and a surgicaltable.

FIG. 9D is an illustration of an arm portion of a manipulating systemshowing example rotational movement degrees of freedom for a forearmportion.

FIG. 9E is a perspective illustration of a portion of the arm shown inFIG. 9D showing a yaw axis, roll axis, and pitch axis.

FIG. 10 is an illustration of an instrument mount and telescoping spar.

FIG. 11 is a plan view of a surgical theater including a plurality ofsystems that each has an asymmetric base.

FIG. 12 is a schematic illustration of example components of amanipulating system.

FIGS. 13A-H are an illustration of aspects of example light featurepatterns.

FIG. 14 is an illustration of an example robot arm.

FIG. 15 is an illustration of an example robot arm.

FIG. 16 is an illustration of an example manipulating system with lightfeatures.

FIG. 17 is an illustration of a user control unit.

FIG. 18 is an illustration of a manipulator with light features.

FIG. 19 is an illustration of a manipulator on a movable manipulatorunit.

FIGS. 20A-20B are illustrations of a medical device that has lightfeatures.

FIG. 21 is an illustration of a medical device that has light featuresaround a telescoping column.

FIG. 22 is an illustration of a medical device system that includes alight feature at a swivel connector.

FIGS. 23A-23B are illustrations of a medical device that has lightfeatures around a joint with multiple degrees of freedom.

FIG. 24 is a flowchart illustration of a method of controlling a medicaldevice having an interface.

FIG. 25 is a flowchart illustration of a method of controlling anintegrated display on a teleoperated surgical system.

FIG. 26 is a flowchart illustration of a method of presentinginformation on a display on a teleoperated surgical system.

FIG. 27 is a flowchart illustration of a method of controlling aconfiguration of a manipulator arm and a handle of a teleoperatedsurgical system.

DETAILED DESCRIPTION Overview

A manipulating system, such as a medical device or medical system (e.g.,teleoperated surgical system that operates at least in part undercomputer control), may include one or more user-friendly features, suchas a light feature (e.g., a light ring), an integral display that may becontrollable to display content to or from a user, a display (e.g., anOLED display) between arm links in a medical device, or a controllablemedical device handle. The system may be positionable in one or morekinematic postures or may include ergonomic features.

An example manipulating system may have two or three (or more)independently movable links that that may move relative to one another.Joints between the links may operate under motor power (“active joints”)or may be unpowered (“passive joints”) so as to be movable by hand. Insome examples, the active joints may be teleoperated active joints thatare controlled by inputs from a user through a user control system.

A modulating system may, for example, include a manipulatable arm, whichmay have discrete jointed links, or a multiplicity of links, or acontinuous construction. The arm may be manipulatable into one or morepredefined kinematic poses. A medical instrument, such as a grasper,cutter, cautery tool, stapler, or camera may be mounted at the distalend of the arm and may be and used to perform a therapeutic ordiagnostic medical procedure. An example manipulator arm may includethree main sections. The first section may be a teleoperated section (a“manipulator”), which may be controlled by a user such as clinician. Amedical instrument may be mounted at a distal portion of themanipulator, and a clinician may move or control the instrument andoperate features of the instrument via teleoperated control of themanipulator and associated equipment. A proximal portion of the firstsection of the manipulator arm may be coupled to a second section, whichmay be oriented or positioned to maintain the proximal end of themanipulator at a desired location in space. The second section may beactive, passive, or both active and passive. The second section may alsobe teleoperated. A proximal portion of the second section may be coupledto a third section, which may for example include a column that extendsgenerally upward from a base. The column may include one or more joints,which may be active, passive, or both active and passive. The columnsection may also be teleoperated. The second section and the third(column) section are optional. The manipulator may be coupled directlyto the column or to the base, or the second section may be coupleddirectly to the base.

A manipulating system (e.g., arm) may actively move into one or morepredefined poses, which may indicate readiness to perform a particularfunction, e.g., “ready to dock to a cannula” or “ready for transportacross the floor” or “in standby and waiting for deployment to anotherpose.” In various examples, system behavior or functionality (e.g.,joint states, light states, user interface behavior) may vary based atleast in part on a pose of the manipulating system.

A manipulating system may be designed to be modular, e.g., a modularmanipulating system may be designed to work as a stand-alone unit,together with or as a part of a larger system, or with another modularsystem (e.g., side-by-side at a table, or across from each other at atable, or otherwise located together). For example, the system 200 shownin FIG. 2A-J and the system 900 shown in FIGS. 9A-C may optionally beconfigured as components in one or more modular systems. In someexamples, two or more modular system may be configured to work in acoordinated manner, or to communicate with each other, or to movetogether, or share sensor information or instructions or otherinformation, or any combination of these configurations. In variousexamples, a modular manipulating system may be on a movable base; or itmay be mounted to a surgical patient or other medical table; or it maybe mounted on a floor, wall, or ceiling of an operating room; or it maybe mounted to another object in a clinical environment. And, inaccordance with various aspects described herein, the base may bemovable with reference to the surgical table, floor, wall, ceiling orother object to which it is mounted.

Aspects disclosed herein are by their nature interoperative because allare generally associated with designs for effective and safe humaninteraction with a teleoperated surgical system. It is contemplated thateven though an individual aspect as described herein may stand alone asinventive, two or more aspects as described herein may be combined asinventive, too. For example, aspects associated with light features maybe combined with aspects associated with controllable displays, and bothmay be combined with aspects associated with ergonomic touch points orcontrollable handles. To avoid prolix description, therefore, aspectsdescribed in one section of this description are not limited totechnical application only within the context of that one section. Forexample, one or more aspects of a light feature described in one sectionmay be combined with one or more aspects of a light feature described inanother section. And, one or more aspects of one feature described inone section may be combined with one or more aspects of one or moreother features described in one or more other sections. For example, oneor more aspects of a light feature may be combined with one or moreaspects of a handle.

Manipulating System Designed on a Human Scale

A manipulating system (e.g., the system 200 shown in FIGS. 2A-2J orsystem 900 shown in FIGS. 9A-E) may be designed to incorporate featuresthat promote and guide human interaction, taking account of humanpsychological and physiological principles. An important psychologicalaspect of human-machine interaction occurs even before a machine istouched. For example, a system may include a manipulator arm (such asarm 204, 904 shown, e.g., in FIGS. 2A and 9A) that is designed by itsappearance to communicate approachability or safety to a human operator,for example as shown in FIGS. 2A-2J, and 9A-9E. In contrast to theexamples shown in FIGS. 2A-2J and 9A-9E, other robots (industrial orsurgical, see, e.g., prior art FIGS. 14-15 ) present sharp edges, pinchpoints, lumps and bumps, and other intimidating features that are lessinviting for human physical interaction and do not appear to be safe orwise to stand next to or touch. For example, even smoothed links presentcertain perceived danger areas that a user is reluctant to touch due toreal or perceived points that might pinch, crush, shear, cut, orotherwise injure, or even due to perceived bulkiness and unwieldiness.

The system shown and described herein may have consistent circular orrounded cross section from the column along the arm and across thejoints until the distal end. (See, e.g., FIGS. 2A-2J, 9A-9E; compareFIGS. 14-15 .) For example, the system may include straight links thatare connected by curved links (e.g., shaped similar to “elbow macaroni”)that maintain the consistent circular or rounded cross section of theother links of the arm. The curved links may be shorter than thestraight links. In various examples, the user's body parts (e.g.,fingers, hands or arms) cannot get pinched or trapped by arm links, dueat least in part due to the configuration of the arm with a consistentcross section and rounded surfaces that space coupled links apart fromone another. Joint range of motion limits may further prevent pinchingor trapping. The consistent and rounded appearance may also communicatea lack of pinch or injury points and a general sense of safety for theuser. (Compare FIGS. 14-15 .)

Some systems may include a column that has a cross section and size thatis similar to the arm portions to further promote the overall humanfactors design principals. (See, e.g., FIG. 2B and FIG. 9A.) The columnportion may include an optional telescoping joint. The column may rotatewith respect to the arm, the base, or both. An optional handle link maybe included at a joint location. (See FIGS. 2A and 2B.) Theconfiguration and location of the handle may convey safety andfriendliness for human physical contact and interaction.

Displays

Various displays may be optionally incorporated into the patient-sideunit, and especially into the arm. For example, LED and/or OLED lightsand light displays of various colors may be used, and they may bedirectly visible, or they may indirectly illuminate another piece orpieces. For example, LED lights may be visible to a person looking atthe patient-side unit, or LED lights may illuminate other pieces orstructures such as translucent rings or bars. Similarly, OLED screensmay be integrated into the unit, such as in a link of an arm or at ajoint between two major links of the arm. Words and symbols, both eitherstatic or animated, both with either static or changing color orbrightness (e.g., hard on/off flash, or softer more gradual flash), maybe variously displayed as clinically necessary or appropriate to conveyinformation such as system, arm, or instrument status. For example, thedisplay or displays may be used to identify an arm or device, to lead auser through a sequence of steps associated with operating thepatient-side unit, to convey joint range of motion limits, to anidentify instrument type associated with the patient-side unit, toindicate the source of a command or operation (e.g., a teleoperatedcommand from a user control unit, or automatic, or manual), to indicatea posture of a device, or to communicate other information.

Light Features

An arm portion of a manipulating system, or other portions, may includeone or more light features. A light feature may be configured orcontrolled to provide information to a user, such as a clinician, aboutthe state of the system or the state of a medical device. For example, alight feature may be at an interface, and may be controlled to provideinformation about the interface, such as information about aconnectivity state, a configuration state, or a movement state. A lightfeature (e.g., light ring) may, for example, be at an interface betweenportions (e.g., at or near a joint between arm links) of a medicaldevice and may be controlled to communicate information about therelative state between the portions (e.g., joint angle, joint positionrelative to joint range of motion, or joint movement state). A lightfeature may additionally or alternatively be at an interface withanother device, and may communicate information about a whether anotherdevice is coupled to the medical device at the interface, the type ofcoupled device, or a control parameter (e.g., which device is receivingor delivering control signals). In some examples, a medical device mayinclude multiple light features, each of which is controlled orconfigured to provide information about a different aspect or state ofthe medical device. Some examples may include an integral display thatmay be configured to be viewable by a user when information ispresented, and to disappear when information is not presented. Anintegral display may, for example, identify a manipulator arm, orprovide information about an operating mode or a phase or status of aprocedure. A display between links or on a handle may be configured toprovide context-sensitive information, or assume a context-sensitiveposition, or “follow a user” so that the handle or display is easilyreachable or viewable to a user or from a specified user referenceposition in two or more different arm positions or configurations.

Handles

A manipulating system may include one or more handles that facilitateoperation of either the system as a whole or a part of the system. Anyof the handles described below may include a display or light feature. Ahandle orientation, or a display or light feature on or near the handle,may be controlled to facilitate use of or interaction with the system bya user. The orientation of one or more handles, or the state of one ormore displays or light features (on or near the handle, or elsewhere ona system), may be coordinated to provide a consistent communication to auser, or may be coordinated based on a state of the system or a mode ofoperation. Handles, lights, and displays may be controlled by aprocessor in a modular system into which they are integrated (e.g., acontrol circuit for the modular system, or a control circuit for aportion of a system, or a control circuit for a joint, or a controlcircuit in or at the handle, light, or display), or by a separated usercontrol system, or separate processing equipment.

Example Systems

FIG. 1A is a plan view illustration of an example medical procedureenvironment that may include a multi-arm manipulating system 100adjacent to a surgical table 101 that may support a patient 103. Asecond manipulating system 200 may also be situated at the surgicaltable 101. The manipulating systems 100, 200 may be free-standing on amovable base, or they may be mounted to a table, floor, wall, orceiling, or they may be supported on another piece of equipment in theclinical environment.

The manipulating system 100 or system 200 may be part of a larger system10, which may include other sub-systems (e.g., fluoroscopy or otherimaging equipment). For example, one or both of the manipulating systems100, 200 may be operatively coupled to a user control system 150 or anauxiliary system 175, or both. The user control system 150 may includeone or more user input devices (e.g., controls) that may be configuredto receive inputs from a user (e.g., clinician). The user control system150 may also include or one or more user feedback devices (e.g., viewingsystem or tactile or auditory feedback) that may be configured toprovide information to the user regarding the movement or position of anend effector, or an image of a surgical area. The auxiliary system 175may, for example, include computer processing equipment (e.g., aprocessor circuit or graphics hardware) or communication equipment(e.g., wired or wireless communication circuits).

FIG. 1B is an illustration of example manipulating system 100. Themanipulating system 100 may include a base 102, a support tower 104, andone or more manipulator arms 110, 111, 112, 113, which may be mounted onthe support tower 104. An instrument 130 (shown in FIG. 1E) may bemounted to an instrument mount 120 on one of the manipulator arms110-113. The instrument mount 120 may, for example, include aninstrument carriage 122, which may be mounted to a spar 124, which maybe a telescoping spar as described below in reference to FIG. 10 . Acannula 133 may be mounted to a cannula mount 126, and the instrument130 may be inserted through a cannula seal in the cannula 133, and intothe patient 103 (FIG. 1A) for use in a surgical or other medicalprocedure. Through movement of the manipulator arms 110-113, theorientation of the instrument 130 may be controlled in multipledimensions, e.g. lateral, horizontal, vertical, angular movements inone, two, or three planes. The system 100 may include one or more lightfeatures 136, 138, 140, 142, 144, 146 at one or more of a variety oflocations on the manipulator arms 110-113 (i.e., at joints between armlinks, as shown).

FIG. 1C is an illustration of example user control system 150. The usercontrol system 150 may include hand controls 155, 156 and pedal controls160, 161, 162. The hand controls 155, 156 and pedal controls 160, 161,162 may be used to control equipment at one or more of the manipulatingsystems 100, 200. For example, portions of a distal end of an instrument130 may be manipulated using the instrument controls. The controls mayinclude haptic feedback features so that a physician may interpretphysical information at the instrument 130, such as resistance orvibration, through the controls. The user control system 150 may alsoinclude a viewing system 165 that may display video or other images of asurgical site.

FIG. 1D shows example auxiliary system 175. The auxiliary system 175 mayinclude computer processing system 180 for processing controls,facilitating communication between the user control system and themanipulating system, or a remote site. The auxiliary system 175 may alsoinclude a display 190, which may show images that the user (e.g.,clinician) is seeing on the user control system 150, a video feed from acamera in the patient 103, or other information. In an exampleconfiguration, signals input at a user control system 150 may betransmitted to the processing system 180 on the auxiliary system 175,which may interpret the inputs and generate commands that aretransmitted to the manipulating system 100 to cause manipulation of aninstrument 130 or portions of a manipulator arm 110. The processingsystem 180 is shown on a cart for exemplary purposes, but may also bearranged in various configurations, e.g., it may be integrated as partof the user control system 150, the manipulating system 100, 200, orboth, or divided between the user control system 150 and manipulatingsystem 100, 200. The equipment may also be provided as software,hardware, or both, on an installed or remote system.

FIG. 1E is an illustration of an example instrument 130. The instrument130 may include a proximal portion 192 that may be configured to coupleto an instrument mount on a manipulator arm, such as the instrumentmount assembly 1002 shown in FIG. 10 . The instrument 130 may alsoinclude a distal portion 194 and an instrument shaft 196 between theproximal portion 192 and the distal portion 194. The distal portion 194may be a cautery tool, cutter, camera, stapler, or other medicallyrelevant end effector. The instrument 130 may be teleoperativelycontrolled via command signals received from a control computer, such asa user control system 150 or auxiliary system 175 to conduct a surgicalprocedure. Inputs may be received from a user (e.g., clinician), and theinstrument 130 may be controlled based on the user inputs.

In an example, the position, orientation, or motion of an end-effectormay follow the motion of the input device coupled to a controller. i.e.,the position and pose of the end effector follow the position and poseof the input device, which may be controlled by a clinician. This may bereferred to as “following mode”, meaning the end effector motion followsthe input device motion. Hardware or software to implement a followingmode feature may be implemented in the system 100 shown in FIGS. 1A and1B, in the system 200 shown in FIGS. 2A-2J, in the system 900 shown inFIGS. 9A-E, and in other similar systems.

FIGS. 2A and 2B are respective perspective and side view illustrationsof example manipulating system 200. The manipulating system 200 may beused with or be part of a medical system, such as the system 100 shownin FIG. 1A, which for example may be or include a da Vinci® surgicalsystem commercialized by Intuitive Surgical, Inc., Sunnyvale, Calif. Invarious examples, the manipulating system 200 may be used in combinationwith the manipulating system 100 shown in FIG. 1B, or with the usercontrol system 150 shown in FIG. 1C, or the auxiliary system 175 shownin FIG. 1D. The same computer or a different computer may control themovement of the arms, joints, lights, or other aspects of the unit, andthe same computer or a different computer may control the manipulatingsystem 200 and other manipulating systems (e.g., system 100 or system900). In some examples, two or more manipulating systems 200, 200′ maybe used together to conduct a computer-assisted teleoperated surgicalprocedure controlled by a clinician (e.g., see FIG. 2F).

The manipulating system 200 may include a base 202, which may beconfigured to roll on the floor, for example, using a plurality (e.g.,three, four, or five) roller balls 203 (hidden under the base 202 inFIG. 2B) or horizontally rotatable caster wheels 203′ (shown in FIG.2C). The rotatable casters 203′ or roller balls 203 (e.g., three or fivecasters or roller balls) may provide a low height profile, may provideomni-directional movement, or both. In an example, five roller balls maybe used, which may provide more stability than three roller balls. Theballs may, for example, be a flange ball transfer available from AshlandConveyor, Ashland. Ohio. The balls may be one (1) inch (2.54 cm)diameter, and may be formed from carbon steel.

In some examples, a user-engageable switch may engage or release one ormore brakes (not shown) on the base rollers, wheels, or casters (or theuser-engageable switch may engage or release other equipment (e.g., adrive motor)) to allow the base 202 to move along the floor, or lock thebase in position, or allow only a defined range of movement (e.g.,movement in a circle or around a circle by locking one or more rollers,wheels, or casters). The system 200 may include an optional touch sensoron an arm 204, on the base 202 or elsewhere, that releases the brake onbase rollers, wheels, or casters to allow the base 202 to move along thefloor. For example, the system 200 may include a foot switch on the base202. In some examples, the system 200 may include a “two-factor” switch(physical or logical) to assure deliberate activation (e.g., the systemmay require simultaneous or sequential engagement of two buttons or twoinputs to release a brake). In some examples, the system 200 may includea “dead man switch” so that the base 202 stops when the switch isreleased. User-engageable switches may be associated with useful systemtransport modes or poses, such as the “IV stand” (mimicking the commonintravenous fluid support stand) pose and transport poses describedbelow. In some examples, the functionality of a switch (e.g., whichfeature or component the switch controls or affects, or the result ofactuating the switch) may vary based upon a pose of the system (e.g., IVstand or other transport pose), or based upon a mode of the system(e.g., following, clutch state, awake state), or based upon anothercondition or state. In various examples, a component of the system 200,such as a handle, touch point, or helm may be or include a user inputfor a brake, brake release, dead man switch, or other switch.

In some examples, the base 202 may include variations in color (e.g.,grey) or material (e.g., ridges, other textures) to visually communicatefoot or hand push-points to a user. For example, a feature or aspect ofan arm (e.g., arm geometry or handle) may invite a user to touch or grabthe arm (as further described below). The base 202 may similarlyindicate a physical touch point for a foot of a user, e.g. a foot pushpoint. The foot push point may, for example, invite the user to use afoot to contact the base 202, based upon the placement (e.g., near theedge of the base), color (e.g., different from main unit color), texture(e.g., ridged or channeled similar to other foot tread features), orappearance of the foot push point.

As shown in FIG. 2B, the base 202 may include a foot push pad 205, whichmay be made of a rugged or high-friction material to facilitatemanipulation of the position of the manipulating system by a foot of auser. The foot push pad 205 in combination with the roller balls 203 orcasters 203′ may accommodate a user pushing against the base 202 with afoot to translate or reorient the base 202 on the floor.

The base 202 may be compact to fit near a table (e.g., a surgicaltable). The compact form of the base 202 may avoid taking up standingspace near the table to be as minimally interfering as possible toclinical personnel, or may accommodate side-by-side positioning of othermanipulating systems by being shaped to allow close spacing (e.g., seeFIG. 2F). In some examples, the base 202 may be symmetric, which mayavoid orientation or configuration problems at the bedside (e.g., seeFIGS. 2A, 2B, and 2F). In other examples, the base 202 may be asymmetricto provide both stability and compactness (e.g., see FIGS. 9A-9B).

The manipulating system 200 may include an arm 204, which may include aforearm 211 that may be coupled at a wrist portion 206 to an instrumentmount 208 and to the base 202. The wrist portion 206 may be configuredto have one or more rotational degrees of freedom, e.g., the wrist mayyaw, pitch, or roll to adjust the position of the instrument mount 208,as shown in FIG. 9E and described in reference thereto. In someexamples, the forearm 211 may roll, and the wrist portion 206 may yawand pitch. The instrument mount 208 may include a telescoping spar 209(see also telescoping spar 1012 in FIG. 10 .)

The manipulating system 200 may include a column 210 extending upwardfrom the base 202. The column 210 may move (e.g., rotate or translate)with respect to the base 202, the arm 204, or both. In some examples,the column 210 may rotate relative to the base 202 at a column-baseinterface. A rotatable column 210 may be particularly useful when thebase 202 is non-symmetric, as shown in FIG. 9A-9C. The column 210 mayoptionally include two or more separate column portions (not shown),which may move with respect to one another.

The manipulating system 200 may also include a helm 213 (also shown inFIG. 8 ), which may for example be positioned at the top of the column.The helm 213 may be at any horizontal rotational angle or verticalrotational angle relative to the base 202. For example, the helm 213 maybe perpendicular to the column 210, or the helm 213 may be inclineddownward from the column 210. In some examples, the helm 213 may be 90degrees or 180 degrees from bedside when the unit is in “neutral”position (that is, generally at mid-range of teleoperated arm DOFs asused during a medical procedure) at the bedside. The helm 213 may alsobe arranged at other angles with reference to the bed. In some examples,the helm 213 may rotate around the column 210 portion of the arm 204 sothat the helm's orientation can be easily adjusted.

The helm 213 may, for example, be arranged and configured to move andextend in an opposite direction from the forearm 211. For example, thehelm 213 may rotate with the elbow 218, or column 210 may include arotatable portion upper link coupled to the arm 204 and that moves(e.g., rotates) relative to a lower link of the column portion on whichthe helm 213 is mounted.

An enlarged perspective view of the helm 213 is shown in FIG. 8 . Thehelm 213 may include one or more buttons 808, 810. Button 808 may, forexample be a power on/off button that may turn power to the system 200on and off. Button 810 may, for example, be a start/stop button that maystart or stop movement of the system, or release the system to allowmovement (e.g., activate or deactivate a clutch). The button 810 may bean emergency start/stop button. In some examples, the helm may include ahandle 804, which may flare out from the helm column mount location 802to provide a wider grab length for the hand of a user at a helm grablocation 814. A light feature 806 may be provided at an interfacebetween the helm 213 and the column 210, or at an interface between thehelm 213 and elbow 218, or both. The light feature 806 may, for example,be a ring that extends part way or all of the way around the column 210.In some examples, the helm 213 may be configured to be sterile. Or, thehelm 213 may be configured to be non-sterile, in which case the helm 213may be positioned relatively low on the manipulating system 200 and awayfrom the sterile field, and so the helm 213 may not be required to bedraped during a surgical procedure (drape not shown).

Returning to FIGS. 2A-2J, in various examples, the arm 204 may beconfigurable to allow for positioning of the instrument mount 208 and anattached instrument 207 with respect to a patient and surgical table, asshown for example in FIG. 2C. In various examples, the arm 204 may becontinuously flexible, or the arm 204 may include a plurality of links,which may be configured to move with respect to one another. Links maybe long or short, and may be straight, or may include a bend, angle, ortwist. The manipulating system 200 may also include one or more elbows,which may be configured with a bend (e.g., at 90 degrees, 45 degrees, or30 degrees) to facilitate relative movement of links with respect toeach other or to the base 202. In some implementations an elbow is aseparate curved link with a joint at each end, and in someimplementations an elbow is integral at one end of an extended straightlink and has a joint at the fre end of the elbow curve. Short or longstraight links may be between discrete or integral elbows. And, a shortor long straight link may have an integral elbow bend at one or bothends. The manipulating system 200 may be configured to provide angularmovement at the interface between two links or elbows, e.g., an elbowand a link may be coupled together at a planar interface, and the elbowand link may rotate with respect to each other around a rotation axisthat is orthogonal to the planar interface, as indicated by arrows. Suchmovement may, for example, be controlled by one or more electric motors,which may be operatively coupled to a control system, such as the usercontrol system 150 shown in FIG. 1C. The system 200 may include anaccess panel on an arm, elbow, or other location to provide access to amotor, a display inside the arm (described below), or other internalcomponents. Example access panels 450, 452 at elbows are shown in FIG.4B. In some examples, the movement of the system 200 may additionally oralternatively be manually controllable, such as by using buttons on thesystem 200 or by manually moving arms, elbows, or other components.

In some examples, the arm 204 may include a combination of links andelbows, which may have a consistent cross section throughout some or allof the arm 204 (with the optional exception of a tapered forearm). Forexample, as shown the manipulating system 200 may include a first link214 and a second link 216 coupled to the first link 214. The first link214 and second link 216 may be coupled to each other and to the column210 by one or more other links or elbows. For example, the manipulatingsystem 200 may include a first elbow 218, which may be rotatably coupledto the column 210. The first elbow 218 may also be configured totelescope (e.g., move upward) relative to the column 210. The system 200may also include a second elbow 220, which may be rotatably coupled tothe first elbow 218. One or both of the links 214, 216 may include abend. For example, the first link 214 may include a first bend 222 at aproximal portion 223 and a second bend 224 at a distal portion 225. Forthe purpose of describing the system 200 shown in FIGS. 2A and 2B, whenconsidering a path along the arm 204 from the column 210 to theinstrument mount 208, “proximal” refers to positions toward the column210 and “distal” refers to positions toward the instrument mount 208.The manipulating system 200 may include a short link 226 between longerlinks. The short link 226 may be coupled to the distal portion 225 ofthe first link 214. In various examples, the short link 226 may includea handle 227 as shown, a display (e.g., see FIGS. 4A and 4B), or adisplay on a handle (e.g., see FIGS. 7A and 7B). The system 200 may alsoinclude a third elbow 228 that may be rotatably coupled to a distal sideof the short link 226. The third elbow 228 may be rotatably coupled to aproximal portion 217 of the second link 216. In other examples, thesystem 200 may include more, or fewer, links or elbows. As shown, theelbows and bends alone (e.g., elbows/bends 224 and 228), or the elbowsand bends with a short link in between (e.g., elbows/bends 224 and 228,with link 226) offset longer links that swing in parallel planes (e.g.,214, 216) from each other. As a result, there is a safety clearancebetween the longer links that the user perceives, and this perceivedsafety promotes human friendliness and approachability, which enhancesuse associated with, for example, the custom poses described below.

In some examples, an outer surface 272 (e.g., metallic, ceramic, orplastic skin) of the arm 204 may cover motors, joints, other componentssuch as wires, control boards, or internal structural components. Motorsmay be in long links, short links, elbows, or any combination thereof.For example, motors associated with joints may be located in thestraight or elbow links and covered with the outer surface 272. Theouter surface 272 of the arm 204 may, for example, be made of thinmachined aluminum, which optionally may be powder coated. In someexamples, an arm may accommodate a visual display or screen, such as anLED display or OLED display, which may be discrete (e.g., see FIGS.4A-4B) or integrated (e.g., see FIGS. 3A-3C).

The manipulating system 200 is designed on a human scale, and isdesigned to incorporate human engineering (human factors; ergonomics)features that promote and guide human interaction, as described above(see section titled Manipulating System Designed on a Human Scale). Forexample, the system 200 may be primarily white, with polished aluminum(e.g., handle 227 on arm 204 or at helm 213) and grey (e.g., inside ofhandle 227, foot push pad 205) accents. The eye is then naturally drawnto the accents, and so the user easily identifies operating features ortouch points at these locations. The manipulating system 200 may includehuman-approachable (friendly; non-intimidating, inviting to be touched)design elements such as a human-compatible arm design. For example, thearm 204 or other elements of the system 200 (such as the column 210) mayinclude smooth, even surfaces, and consistent cross-sectional dimensions(e.g., 10-14 cm diameter), which may extend through elbow portions, allof which may indicate safety—even attractiveness to be touched—to auser. In an example, an arm includes consistent cross-sections along acurvilinear axis. The arm 204 may, for example, have no pinch points andno mechanical shearing relationship between links, which may becommunicated by the consistent cross section, smooth surfaces, relativeconfiguration of bends, links, or other arm portions. In contrast, FIGS.14 and 15 are example illustrations of prior art robot arms that areless user-friendly than the example manipulating arms shown in FIGS.2A-2J and 9A-9C. FIG. 14 shows a system with an arm 1402 that does nothave a consistent cross-section and perceived hand/arm/finger pinch andcrush points as the links move. FIG. 15 shows a system that has a shearpoint where a first link 1502 meets a second link 1504, as well asintimidating exposed cables, abrupt edges, potentially injurious exposednuts and bolts, etc.

The system 200 may include one or more light features, which may beconfigured or controlled to convey information about the operation orconfiguration of the system 200 and/or its components. A light featuremay be located at a constrained (i.e., a joint) or unconstrainedinterface between two physical objects. Alternatively, a light featuremay be located on a single object, either projecting from, recessedfrom, or integral with one or more of the object's surfaces, which maybe flat, curved, or otherwise shaped or textured. For example, a lightfeature may extend part way or all of the way around an interface andconvey information about the interface. The system 200 shown in FIG. 2Amay include a first light feature 236 at the interface (a joint) betweenthe column 210 and the arm 204 (e.g., at the first elbow 218), a secondlight feature 238 at the interface (a joint) between the first elbow 218and second elbow 220, a third light feature 240 at the interface (ajoint) between the second elbow 220 and the first link 214, and a fourthlight feature 242 at the interface (a joint) between the third elbow 228and the second link 216. A fifth light feature 244 may be provided onthe instrument mount (object surface). Additionally or alternatively,light features may also be provided at other locations, such as on thebase 202 at an unconstrained interface with the floor (e.g., wheels,roller balls) or with the column 210 (e.g., the curved surface of anintegral base and column in FIG. 2A; joint in FIG. 9A).

FIG. 2C is an illustration of a manipulating system 200 positioned overa surgical table 250. The first link 214 and second link 216 have beenmanipulated to position the instrument mount 208 over a surgical site ofa patient 251. The example manipulating system 200 shown in FIG. 2Dincludes an additional short member 252 between the first elbow 218 andthe column 210. FIG. 2C illustrates the minimum impact of an individualsystem 200's volume at the bedside, which allows clinical personnel toeasily access the patient 251 while the system 200 is in use (see the“hybrid” surgery described below), and which allows one or moreadditional systems 200 to be placed nearby.

FIG. 2D is an illustration of a manipulating system 200 being used in asurgical procedure. An instrument carriage 282 may be mounted to a rail280 on the spar 209. A proximal portion 284 of the instrument 207 may bemounted to the instrument carriage 282. The arm 204 may be oriented andpositioned to situate the instrument 207 through a cannula 254 and intoa patient. The cannula 254 may be coupled to a cannula mount 286 on thespar 209. The instrument carriage 282 may be translated (e.g., moved upor down) on the rail 280 (e.g., using a lead screw, not shown), whichmay insert (advance) or retract (withdraw) the instrument 207 withrespect to the cannula 254 or patient. The arm links 214, 216 and elbows218, 220, 228 may be sized, shaped, and configured to position theinstrument mount 208 over a subject (e.g., patient) while allowing spacefor a user to access the subject, as shown in FIG. 2D.

A manipulating system (e.g., system 100 or 200 or 900) may have or use asoftware-constrained remote center of motion for a surgical instrument.The center of motion may be maintained at least in part by controlling aspar orientation (e.g., orientation of spar 209) with reference to aforearm link (e.g., link 216) as a user (e.g., clinician) operates oneor more user inputs on a user control system (e.g., user control system150).

Optionally, a second instrument 256 may be inserted through a secondcannula 258 and controlled manually by a clinician (as shown), or it maybe controlled by a second manipulating system (e.g., where two systemsare positioned at a surgical table as shown in FIGS. 2F and 9C).

As illustrated, therefore, in some examples a manipulating system (e.g.,system 100 or 200 or 900) may be used in a “hybrid” minimally-invasiveprocedure in which a user (e.g., clinician) controls one or moreteleoperated instruments 207 (e.g., via a remote control device), andthe same user or a different user controls one or more manualinstruments 256. In some examples, light features and display features,as described above and below, may contribute to a “user friendly”telesurgical environment, e.g., when the clinician is located next tothe teleoperated arm.

FIG. 2E is an illustration of an arm portion of a manipulating system200 extending across a surgical table. A distal portion 260 of the armmay include a narrowed portion 262 (e.g., may have a tapered or narrowedwidth or cross-section), which may provide extra operating space for aclinician 264 to work over a patient. The narrowed portion 262 may forexample be on a second link 216 of the arm 204. For example, the arm 204may include a tapered or thinned forearm link that provides improvedaccess (relative to a non-tapered forearm) to a patient (see, e.g.,FIGS. 2E, 2F, 9A, 9B, 9C). The narrowed portion or taper on the arm 204may promote clinical efficacy by allowing better access to the surgicalfield and to the patient in general. The narrowed portion or taper mayprovide clinician with more space for hands or tools. The narrowedportion or taper on the arm 204 may also allow a second arm to work moreclosely without collision. In various examples, a taper on an arm may beabrupt, rapid, gradual, abrupt or rapid and then gradual, or abrupt orrapid, and then straight. The taper may be cross-sectionally symmetric,horizontally symmetric, vertically symmetric, or any combinationthereof.

The arm 204 may also include a display 266 (e.g., an OLED display,discussed below), which may be on the short link 226 between the firstlink 214 and second link 216, or a touch point (also discussed below),which may be on an elbow. The display 266 may be configured to provideinformation to a user at the table, and the touch point may beconfigured to receive an input from a user at the table, both of whichmay promote clinical efficacy by providing information and input optionsto a user who is working on or near a patient and yet do not interferewith the overall human-engineered form of the arm itself.

In some examples, the base 202 may be powered (e.g., via power suppliedto casters 203 or wheels 203′ or via power delivered to one or moreother objects configured to engage a floor beneath the base 202). Thebase 202 may be computer-controlled via inputs on the base 202, or itmay be controlled using a secondary device (e.g., handheld device), orthe base 202 may be teleoperated (e.g., using user control system 150).The base 202 may be controlled (e.g., casters or wheels locked) to bestationary while docked to a cannula (e.g., during a surgicalprocedure). Alternatively or additionally, the base 202 may move whiledocked to cannula. For example, the base 202 may move when the system200 is controlled by inputs from a user control system (e.g., during afollowing mode). The base 202 may be moved in response to specificinputs from a user to move the base 202, or the base movements may bedetermined based on a combination of conditions and inputs (e.g., theposition of the base 202 may be adjusted to facilitate or enableorientation of a cannula or the arm). Information on movement of thebase while docked or during following is found, e.g., in U.S. PatentApplication Pub. No. US 2017/0181801 A1 (filed Feb. 15, 2015), which isincorporated herein by reference.

Movement of the base 202 may allow more freedom of movement for system200 in general, e.g., provide extra range of motion (ROM) for the arm204. The base 202 may be moved, for example, as an arm 204 approachesend of range of motion.

In some examples, the system 200, or user control system 150, or anotherdevice may issue a warning as an arm joint approaches a range of motionlimit as the base 202 moves. In some examples, an arm joint may lock asROM limit occurs as the base 202 moves. For example, the system 200 maylock a joint at or near its ROM limit. In some examples, the system 200may lock all arm joints as the base 202 moves.

FIG. 2F is an illustration of two manipulating systems 200, 200′positioned side-by-side over a surgical table. The systems 200, 200′ mayinclude design features that allow two or more manipulating systems towork and fit closely next to each other. For example, the systems 200,200′ may be sized and shaped to sit side-by-side so that the arms 204,204′ on each system may reach across the surgical table 250 to provideinstrument mounts 208, 208′ that may be mounted to correspondingsurgical instruments 207, 207′ for use in a surgical procedure. As shownin FIGS. 2F and 9C, for example, two or more forearms may be positionedclose together, which may, for example, allow cannulas to be placedclose together.

In some examples, the base 202, 202′ of each system 200, 200′ may besized and shaped to allow the systems to be situated close together.Design features to enable side-by-side operation may include forexample, a tapered forearm and a small base area, as well as(optionally) a column and an upwardly-extending portion the arm thatstay generally within the vertical boundary of the base, with theoptional exception of portions that extend over the table. In someexamples, the systems 200, 200′ may be configured to work side-by-sideon the same side of the table (e.g., as shown in FIG. 2F) or on oppositesides of the table (e.g., as shown in FIG. 9C). The systems 200, 200′may include narrowed portions 262, 262′ (e.g., forearms), which mayallow instruments 207, 207′ to be placed close together and give aclinician maximum space to work. In a clinical environment, a steriledrape (not shown) is typically placed over a manipulator arm. In someexamples, bases 202, 202′ may be sized and shaped to take about the sameamount of space as a person standing beside the table.

In some examples, the manipulating systems 200, 200′ may be bothcontrolled by the same user control system, such as the user controlsystem 150 shown in FIGS. 1A and 1C. In other examples, eachmanipulating system 200, 200′ may be controlled by a separate usercontrol system, or one or both of the manipulating systems 200, 200′ maybe controlled by one or more of a user interface, buttons, or tactileresponse (e.g., applying forces to move the arm) or other controlfeatures on one or both of the systems 200, 200′. In some examples, two(or more) units may communicate with each other, e.g., via wirelesscommunication (such as Bluetooth, Zigbee, Z-Wave. Wi-Fi, cameras,acoustic, or light) or wired (e.g., electric, optical) communication.

The portions of bases 202, 202′ of the manipulating systems 200, 200′may be sized, shaped, or configured to allow two or more manipulatingsystems to work and fit closely next to each other (or a close to aperson, or other equipment) on the same side of the table, or onopposite sides of a table. In some examples, the base may be asymmetric(e.g., see base 902′ in FIGS. 9A-9C). Asymmetry may allow a system to belocated closer to the table than if a symmetric base is used. Forexample, for the same arm geometry, an asymmetrical base may provide alonger reach across the table, e.g., because an asymmetric base mayaccommodate a larger overhung (cantilever) load, e.g., by providingcounterweights in an asymmetric rear portion of the base.

In some examples, a base 202, 202′ may be omni-directional (e.g., seeFIGS. 2A, 2B, 2F) so that it is orientation-agnostic for operation. Forexample, omnidirectionality may permit set-up in any direction. The basemay, for example, be symmetric (or nearly so). Portions of a system maybe centered around kinematic base (e.g., base of column portion may becentered on the base). In some examples, a base may be circular (see,e.g., FIGS. 2A, 2B. 2F) when viewed from the top, or the base may be apolygon (e.g., a pentagon, hexagon, heptagon, or octagon), or mildlyoval, in a way that provides the same benefits as a circular, symmetricbase. A base having a combination of circular, polygon, or ovularcontour is also possible.

In some examples, the omnidirectionality (e.g., circular, symmetricand/or polygon) of a base avoids forcing clinical personnel to makedecisions that trade off benefits and disadvantages of asymmetric bases,such as requiring a cannula be approached from a specific direction, ordecisions that bases be placed at certain locations with reference tothe patient or the table because of their shape. While the compactcircular/symmetric/polygon (or combination thereof) base offers setupadvantages, it requires the base to be heavy enough to providestability, and the increased weight requires more power to drive thecart.

As shown in FIG. 11 , in some examples a base may be partiallysymmetric, but with an asymmetric portion that provides added clearancefor personnel or objects. For example, a base 1102 may be circular (asshown) or rounded or polygon-shaped on a side 1104 that faces a table1108 (e.g., surgical table), with a section 1106 removed on the oppositeside 1110 to provide more foot room for a person or clearance for otherobjects. In another example, a base 1112 may have a side 1114 with aportion removed adjacent to the table 1108 to allow a closer position tothe table 1108, and an opposite side 1116 may be circular or rounded orpolygon-shaped.

In some examples, a first base 1120 may have a cutout section removed ona first side 1124 (e.g., flat surface) to provide a narrower footprint,and a second base 1122 may have a cutout section removed on a side 1128(e.g., a flat surface) facing the first side 1124 of the first base1120, which may enable the two systems to be placed closer together sideby side than if the bases 1120, 1122 were symmetrical on adjacent sidesof the bases. In some examples a second side 1126 of the first base 1120and a second side 1130 of the second base 1122 may also include a cutoutsection, which may allow interchangeability of bases in side-by-sideconfiguration (e.g., in FIG. 11 the bases could be interchanged, i.e.base 1120 may be moved to be closer to the end of the table 1108 andbase 1122 may be moved to the other side of base 1120, to be furtherfrom the table 1108, and the bases may still be placed so that columns1132, 1134 are close together as shown). Alternatively, one or both ofthe non-opposed sides 1126, 1130 of the bases 1120, 1122 may not be cutout, which may provide additional stability due to wider weightdistribution or wider support from casters or feet at the bottom of thebase.

Predefined Poses

The manipulating system 200 may assume predefined arm poses or motions,which may correspond to different operating modes. For examples, themanipulating system 200 may assume one or more of the predefinedkinematic poses shown in FIGS. 2G-2J. The poses may include, forexample, factory-defined pose, or a pose that is determined by thesystem by using one or more inputs and a processor in the system 200 orthat is communicatively coupled to the system 200 from outside thesystem. The poses may also include user-defined poses that may be storedin memory, such as custom poses defined by a clinician or other user.Definition of a pose by a user or automatically by a system may allowfor customization of a manipulating system to a particular environment.For example, a pose may be determined based at least in part by a heightof a surgical table, a location of one or more objects, or a location ofa display on the system to make the display more visible to a user.

In various examples, the predefined poses may be user-modified, orlocked, or protected via security feature, such as a password orbiometric. For example, movement of a portion of the system (e.g., alink or joint location) may be lockable or unlockable by using abiometric input, such as a fingerprint, retinal scan, voice recognition,or by using a password (typed, written, or spoken), or based upon aninteraction with buttons, touch points or other inputs (e.g., a releasecode may include a combination of inputs).

A pose may indicate to a user when a manipulating system is ready for afunction to be performed, and in this way one or more poses establish avisual language communication to the user. For example, when an armassumes a specific pose, the pose may indicate that the system is readyto perform a function associated with that specific pose, as furtherdescribed below in reference to the examples shown in FIGS. 2G-2J. Theposes act as a visual language interface through kinematic pose tocommunicate information to a user, such as “I [the system] am ready foryou to do task X” or “You [the user] should do task Y now”.

FIG. 2G is an illustration of a manipulating system 200 in an “IV stand”kinematic pose. In this position, the manipulating system 200 may besimilar to the form factor of an intra-venous (IV) fluid delivery stand,which is typically familiar to clinical staff and may provide a user 269with a familiar size and shape for physical interaction with the system200, e.g., for the user 269 to navigate around the system 200, or movethe system 200 by grabbing the arm 204. The IV stand pose may provide acompact configuration for transport, or for stowing when the system 200is not in use. This is another example of human engineering thatprovides a user-friendly and approachable appearance because it issimilar to an object which the user is already familiar and comfortablewith. In the IV stand kinematic pose, the first link 214 extends upward,and the second link 216 extends downward, with elbows and interfacesappropriately oriented and positioned to provide the compact form factorshown in FIG. 2G. In some examples, in the IV stand kinematic pose, thearm 204 may be entirely or mostly contained in a vertical space (e.g.,vertical cylinder or prism) defined by the shape of the base outerperimeter. A configuration in which the arm 204, or both the arm 204 andthe instrument mount 208, is entirely or mostly contained within avertical space defined by the base 202 may reduce collisions (or therisk of a collision) as the system 200 moves or is moved, because thebase 202 will hit a vertical object (e.g., wall) before the arm 204 orinstrument mount 208 hits the object. In some examples, two or more armlinks may fold back on each other, so that the arm links stow compactlyover the base 202. The spar 209 and instrument mount 208 (at the distalend of the arm 204) may be positioned mostly or entirely over the base202. In some examples, an optional touch sensor on an arm or the base(e.g., foot switch) releases brakes on base wheels to allow the base tomove along the floor.

FIG. 2H is an illustration of a manipulating system 200 in a “ready todrape” kinematic pose. The system 200 may show the word “drape” on anindicator (e.g., integrated display or OLED display) when the system 200is in the drape pose to further communicate the ready-to-drape kinematicstate to the user 269.

In response to a drape command or other input or condition, the system200 may transition to the ready-to-drape pose, i.e., extend an arm foreasy draping (covering portions of the system 200 with a sterile drape)by a user. For example, to facilitate draping, the system 200 may movethe distal end of the arm or a drape mount 268 to a point high enough topreserve sterility (e.g., so that the drape does not fall below thelower boundary of a defined sterile field, such as the surgical tabletop if the area below the table is not part of the sterile field) andyet low enough to allow the user to reach and drape the arm. In anexample, in the ready-to-drape pose the arm 204 may be oriented andpositioned to place the distal end of the arm 204 or a drape mount 268at an appropriate height so that a user 269 may reach the drape mount268 to couple a surgical drape to the drape mount 268. In the exampleshown in FIG. 2H, the arm 204 is positioned and oriented so that thefirst link 214 and second link 216 are coaxially aligned and extendingupward at an angle less than 90 degrees to present the distal end of thearm or the drape mount 268 at an appropriate height for draping with asterile drape. Likewise, the instrument mount 208 may be generallycoaxially aligned with the first link 214 or with both the first andsecond links 214, 216 in order to facilitate easy draping with a steriledrape.

In some examples, the system 200 may lock all the joints in theready-to-drape mode, or the system 200 may lock all joints except one,such as shoulder joint 270 at the top of the column 210 to enable armheight adjustment. In some examples, the free (not locked) joint isgravity-compensated to allow user 269 to easily adjust height. Ingravity compensation, one or more motors or springs counteract gravityforces while the arm 204 is either stationary or moved by hand so thatthe arm 204 acts as if weightless. In some examples, the free joint maybe braked (automatically or in response to user input) to hold thedesired pose after the height adjustment. In some examples, a system maylock all joints except a lower shoulder joint 270 and allow a user tomove the arm 204 vertically down for draping, and the system may thenoptionally return the arm to a previous higher vertical position after adrape is coupled to the drape mount as determined by a sensor or userinput.

The system 200 may accommodate different user heights. For example, thearm 204 or drape mount 268 may be positionable higher or lower to enableconvenient reach by a user.

In some examples, the system 200 may, automatically or through a guidedprocess, assume a specific (e.g., personalized or predefined) drapekinematic pose or drape height for an individual user, e.g. a pose basedon a stored user height, stored user preference, or sensed userinformation. The system 200 may, for example, recognize the height ofthe user, e.g., using visual scanning camera, pre-stored or one-timeentered height information. In some examples, the system 200 may learn acomfort height for an individual user, store the height, and then moveto the user's comfort height the next time the individual user commandsthe ready to drape pose. As a further example, the system 200 operatedby a first user extends the arm to a first ready to drape posecustomized for or preferred by the first user, and the system 200operated by a second user extends the arm to a second ready to drapepose customized for or preferred by the second user.

In some examples, the system 200 may assume different arm positions(e.g., angle with reference to the floor) for the drape pose based on aposition of the handle 227 or helm 213 (both shown in FIGS. 2A and 2B).For example, rotating the handle 227 or helm 213 upward with respect tothe column 210 or arm 204 may correspond to a higher drape position, androtation of the handle 227 or helm 213 downward may correspond to alower drape position.

In some examples, the system 200 may determine a push configurationbased on the arm pose that was used during transport (either the IV posedescribed above or the transport pose described below) or on a handle orhelm position used during transport, and then determine a drape posebased on the push configuration. For example, the system 200 maydetermine a height parameter of a person who moves the system 200 in thetransport pose. If a relatively tall person moves the unit in thetransport pose, the system 200 may subsequently move the drape mount toa relatively higher height (e.g., configure the arm higher), and if arelatively short person moves the unit in the transport pose, the system200 may subsequently move the drape mount to a relatively lower height(e.g., configure the arm so the distal end of the arm or the drape mount268 is lower). In some examples, a feature that determines a ready todrape pose height based on a prior interaction (such as transport) mayhave a time-out, so that the feature is applied only if the ready todrape pose is commanded within a predefined time (e.g., 10 minutes) orother limit (e.g., system has not been placed in standby) after theprior interaction (e.g., after moving in the transport pose).

FIG. 2I is an illustration of a manipulating system 200 in a “ready todock” kinematic pose. An indicator (e.g., integrated display or OLEDdisplay) may show the words “ready to dock” or a corresponding symbol orindication when the system 200 is in the ready to dock pose to furthercommunicate the ready to dock kinematic state to the user.Alternatively, an indicator such as “standby” may be similarlycommunicated when the system 200 is not ready to dock. The “ready todock” pose may be used when the system 200 is deployed for entry into asterile surgical field and subsequent docking to a cannula inserted intoa patient on a surgical table. For example, in the ready to dock pose,when the system 200 is adjacent a surgical table, the arm 204 may extendover the surgical table to position the instrument mount 208 above asurgical site of a patient 251 as shown. In the ready to dock pose, theforearm may be generally horizontal. As described above with respect tothe ready to drape pose, the system 200 may lock some or all but one ofthe joints and enable a user to manipulate one or more joints to movethe forearm or instrument mount 208 to a user-determined height, e.g.,to position the forearm above a surgical site on a patient 251 on atable in order to effectively dock to the cannula, or position theinstrument 207 for use, to provide the desired instrument range ofmotion during surgery. The system 200 may compensate for gravity orprovide assisted moving as described above, and it may lock joints afterexpiration of a period of time or in response to a user command.

FIG. 2J is an illustration of a “ready to transport” or “transport”kinematic pose. An indicator (e.g., integrated display or OLED display)may show the words “ready to transport” or a corresponding symbol orindication when the system is in the ready to dock pose to furthercommunicate the system state to the user. Alternatively, an indicatorsuch as “standby” may be displayed to the user to indicate the system isnot in the “ready to transport” or “transport” state. In the descriptionthat follows, the “ready to transport” pose as described also applies tothe “transport” pose used for moving the system.

In the ready to transport pose, the arm may be placed in an pose thatfacilitates easy transport of the system 200. For example, the arm 204may be positioned and oriented to provide the user 269 with a convenientpush point or push pad 205 for transporting the manipulating system 200by pushing it. For example, in the transport pose, the arm 204 mayextend at least partially back and present an arm, elbow, or handle toprovide a convenient grab point or hand push point. In some examples,the system 200 may be pushed like a shopping cart by using the extendedarm or elbow or handle.

In some examples, two or more arm links may fold back on each other, sothat the arm links stow compactly over the base 202. The upper arm linkmay extend past the proximal end of the lower arm link, which mayposition the spar 209 and instrument mount 208 (at the distal end of theupper arm link) on the other side (relative to the column 210) of thelocation where the lower arm link proximal end joins to the verticalcolumn link.

In some examples, the grab point or hand push point may be outside thevertical boundary of the perimeter of the base to provide clearance forthe user's feet while walking. Even though the push or grab point isoutside the vertical boundary of the base, the system 200 may still bein a relatively compact configuration, for example with most of armwithin the vertical boundary of the base, and a push portion extendingoutside of the boundary. In this way many of the advantages of keepingthe arm within the base's vertical boundary are preserved, and a smallexception is made so that the user is provided a comfortable touch pointfor moving the system. In one example use, a user places the system inthe transport mode to move it relatively longer distances (e.g., downhallways and through doors) and places the system in the IV mode to moveit relatively shorter distances when obstacle clearance is mostimportant (e.g., near a patient operating table).

In an example, the first arm link 214 may be positioned and oriented toextend back toward the user 269, and the second arm link 216 may bepositioned and oriented to extend forward to provide a compact transportform factor, while also providing a grab or hand push point. As shown,the grab or hand push point is at the interface between first arm link214 and second arm link 216, at elbows/bends at this location,optionally separated by a short link as described above or a handle asdescribed below. The rest of the arm and instrument mount remain withinthe vertical boundary of the base, which may be of various shapes asdescribed.

In some examples, the height of the grab or push point may beadjustable, with motion assist or gravity compensation, as describeabove in reference to the ready to drape pose. In an example thatincludes a handle 227, as shown in FIGS. 2A and 2B, the orientation ofthe handle 227 may be adjustable and lockable by the user, or it may beautomatically lockable (e.g., after expiration of a predefined timeperiod with no movement) as described above. In some examples, motorizedwheels or roller balls in the base 202 may assist with transport, asdescribed above.

Various command schemes may be used to manipulate a system into a pose.For example, a system may assume a pose using an automatic process(e.g., automatic posing), or a system may assume a pose using astep-by-step guided process, or system may assume a pose using amanually-assisted process. A manipulating system may be configured toautomatically assume a pose, for example, based at least in part on areceipt of a command from a user interface or buttons on the system, orbased on a receipt of a command from a separate user control system. Ina commanded posing process, a manipulating system may move into a posein response to a user command (e.g., via a user control system, buttons,screen interaction, or voice command). In an example guided process, auser may manually move the arm into a pose through one or more guidedsteps. In a guided process, a system may present guidance to a user byusing audible word commands (e.g., presented on a display or playedthrough a speaker), or by using visible light feature indications (e.g.,a light feature at a joint to be lighted, or may be lighted in a mannerthat indicates a direction of movement), or by using a combination ofaudible word commands and visible light feature indications (e.g.,“Straighten the elbow joint, which is flashing blue”). A system may alsoassume a pose through an unguided process. For example, a user maymanipulate the system into a pose. In some examples, a system may applyfuzzy logic, e.g., when a system is manipulated into a configurationthat approximates or resembles a pose (e.g., joints are within aspecified amount, e.g., 5 degrees or 10 degrees of a certain pose), thesystem may automatically, or in response to a user input or to a userquery response, move the rest of the way into the pose (i.e., the systempredicts the user's intended pose, and then implements the pose). Insome examples, a system may guide a user to a pose by limiting one ormore of available degrees of freedom (e.g., mobility of joints) to leadthe user through steps that progress toward a pose. In some examples, amanipulating system may assume a pose based upon satisfaction of acondition (e.g., no instrument docked and user control system indicatesready for docking). For example, responsive to satisfaction of acondition, a system may automatically assume a pose, or it may output aquery to a user regarding a pose (e.g., “Assume dock pose now?”).

Commands may be received, for example, via a manual input device, voicecommand, or electronic command from another device (e.g., from controlconsole, remote control, or other handheld device). Manual input devicesmay include one or more buttons, force-sensitive areas, touch screens,or tablet controllers, any of which may be coupled to or separate fromthe unit (e.g., handheld), and which may communicate with motors or acontroller via wired or wireless communication technology. Manual inputdevices may, for example, be on the helm or on one or more arms, elbows,or handles. In some examples, translation to a pose may be initiated bytouching the arm at one or more key points (e.g., force sensors orbuttons), or translation to a pose may be initiated by pulling orpushing near a joint or at a handle. In some examples, movement of linksor joint positions may be limited to specified directions or ranges ofmotion, e.g., to assure progress toward a specified pose.

In some examples, commands may be received through a gesture sensingdevice (e.g., Leap® controller by Leap Motion, Inc., San Francisco,Calif.). In some examples, a user's gesture may indicate a direction ornature of movement. For example, a predetermined user gesture maycorrespond to a specific action, position, orientation, or pose of thesystem or a system component. In some examples, a user may draw a shapein the air, and the manipulating system may move into the drawn shape,or a shape interpreted from the drawn shape as described above.

In some examples, one or more commands may be received using anaugmented reality device or feature. For example, a system may includeor receive input from a software widget that may be manipulated by auser.

In some examples, a manipulating system may include one or more forcesensors for receiving input. Force sensors may, for examples, includecapacitive sensors or pressure sensitive sensors. In some examples, asensor or sensor system may be both capacitive, to confirm that a personis interacting with the sensor, and pressure sensitive to receiveinformation that constitutes a command.

In some examples, the manipulating system may apply logic or analgorithm to the receipt or execution of a command. For example, amovement command may be available and active at a specified time or in aspecified condition, e.g., during a sequence, or when a particular modeis enabled. In some examples, a system may require two or moresequential or simultaneous inputs to verify intent to deliver a command.For example, a system may require two hands on the machine, or twopersons interacting, a restriction which may for example avoidinterpretation of an inadvertent touch or force event (e.g., leaning ona system or brushing past a system) as a command, i.e., application oflogic requiring a combination of two or more commands may avoid a riskof an inadvertent bump that triggers a movement. In some examples, toeffectuate a command a system may require a first input to initiate acommand receiving mode and a second input that includes the actualcommand. In some examples, a system may require two types of inputs,e.g., voice and a button.

In some examples, features on or associated with a manipulated system,such as one or more lighted rings or displays, may indicate which linksto move or joint positions to change, or where on the links or joints toimpart movement, or the direction or type or sequence of movement. Forexample, a particular light feature (e.g., light ring) may light up toindicate that a joint position should be changed, and an aspect of thelight feature, such as a lit portion or an animation, may indicate adirection of movement (e.g., to indicate to change a linear orrotational joint position in a particular direction as indicated by ananimation that may extend along or around the joint).

As mentioned above, in some examples, a manipulating system may applycontext-aware or predictive functionality. For example, an operation maybe enabled based at least in part on context. In some examples, a systemmay predict a desired operation (e.g., movement into a particularkinematic pose) based on one or more inputs, software, or application ofinputs to a model. Inputs for context-aware or predictive functionalitymay, for example, include the current pose of the unit, a dock state,button or switch activation, actuation or state of a break-away clutchfeature (e.g., a state of a button or switch that when activatedreleases a brake and allows an arm or joint to move). In some examples,a system context-aware or predictive functionality may determineavailability of kinematic null state adjustment (option for manualadjustment of one portion of the unit without moving another portion ofthe unit, e.g., lock instrument and mounting components but allow somemovement of arms or joints), or be based at least in part upon anavailability of a kinematic null state adjustment (e.g., null stateavailability may form part of condition).

In some examples, the system in a particular pose may indicate that itis in a corresponding mode of operation. For example, the functionalityof a manipulating system, software algorithms, responses to input,mobility of arms or joints or base, handle position, or lightingbehavior may vary as a function of the assumed pose.

In some examples, a particular pose or system state may be indicated oridentified by lighting on rings, or by a presentation on an integrateddisplay that appears to disappear when not active, or both. For example,a color, or flashing lights, or animation, or other waveform mayindicate that a pose or system state has been achieved or assumed, or isactive. In some examples, a color or combination of colors may beassociated with a particular pose. In some examples, a pose may beindicated or identified by sound (audible) output. For example, a systemmay generate a sound, such as a chime or ascending tone sound, when aunit has assumed one of the poses. In some examples, a system maygenerate a different sound for each pose. In some examples, a system maygenerate a sound (e.g., descending tone sound) when a unit has left apose. Or, a system may generate a particular sound when a pose cannot beachieved, for example because of an obstruction or a configuration issuesuch as a joint ROM limit. In some examples, a pose, mode, or otherstate may be indicated by sound and light outputs together in order toparticularly draw a user's attention to the system's communication.

In some examples, a system may perform an automatic docking procedure inwhich a manipulator arm is docked to a cannula that is inserted into apatient. For example, a system control base first motors to move andapproach a destination (e.g., a position adjacent a surgical table). Inan example, the system may enter a compact pose (e.g., IV stand pose orother pose with an arm retracted) for this approach step to reduce thepotential for a collision during the approach, and then the systemextends the arm at the end of the approach to another pose, such as aready to dock pose. In another example, a system may first move into aready to dock pose and then move approach the destination, optionallymaking adjustments during the approach to avoid a collision or to assureproper clearance from operating room obstacles. In some examples, asystem may “roll up” to a surgical table after it has been placed, orplaced itself, into a ready position or orientation (e.g., ready to dockpose). The system may process sensor input (e.g., motion sensors orforce sensors) to avoid collisions, or to preserve or obtain properspace or clearance with other objects. In some examples, a system mayreturn to a defined home location, e.g., in response to a discrete“home” command or in response to completion of a procedure, such as whenthe arm is undocked from a cannula and is moved to an IV stand pose.

Displays

FIG. 3A is an illustration of an integral display 302 on an arm portionof a manipulating system. The integral display 302 may be provided on aportion of the manipulating system 200 shown in FIGS. 2A-2J, asillustrated in FIG. 3A. The integral display 302 may also be provided onan arm of the system 100 shown in FIG. 1B.

In FIGS. 2A, 2B, 2D, and 2H, the integral display is shown located onthe arm 204 (e.g., on the second arm link 216), but the integral display302 may be anywhere on the device (e.g., on the column, the base, on anelbow, on any arm or link, or between major links or arms), and a systemmay include two, three, or more integral displays, which may each showthe same information, or which may work in concert to each showdifferent information, which may for example, be relevant to aparticular part of the system (e.g., relevant to a particular arm linkor relevant to an instrument associated with an arm) or relevant to aviewer having a particular perspective (e.g., field of view). Theposition of the integral display 302 is only limited by space to mount adisplay device, such as an LED display, which may, for example, bemounted in an internal space inside the arm 204.

In some examples, the integral display 302 in the arm 204 may be acontinuation of the surface contour of the portion of the arm 204 (e.g.,arm link) in which the display 302 is located. And in some examples theintegral display 302 may “disappear” or become “invisible” (i.e., noteasily perceived as being a display) when the display 302 is notoutputting information to the user. The ability of the display 302 to“disappear” into the arm 204 may provide a cleaner user experience(relative to a display that does not disappear), and it further enhancesthe human-engineered, user-friendly, ergonomic, and approachablecharacter of the arm 204. For example, the display 302 may appear toexist and output information when the display 302 is needed, and thedisplay 302 then appears to not exist when the display 302 is not neededand no information is output. In some examples, the system may provide arelevance-based display, which may be visually present only during thetime the displayed information is relevant. In this manner, the display(i.e., the manipulating system) may provide a user with only theinformation the user needs or wants at a particular time and at alocation relevant to the user. The capability of a display to disappearwhen not needed also avoids taking up visual real estate on the arm oravoid the visual “noise” or clutter of a permanently visible screen,which may provide for a better user experience by only being presentwhen the user needs it and otherwise not visually distracting the user.

An integral display 302 may be formed in the surface of the arm 204, asshown in FIG. 3A. For example, a grid of very small holes 304 may beformed in the aluminum surface 306. A hydrojet process, for example, maybe employed to form the grid of holes 304. The hydrojet-formed holes (orother types of holes) may be small and practically not visible to thenaked eye, but they may allow light to pass through so that the light isvisible to a viewer of the display 302. In some examples, an LED panel(not shown) may be installed behind the grid of holes 304. A display onthe outside of the surface 306 may be created when light from the LEDpanel shines through the holes. Portions of the LED panel may becontrolled to be lit or dark to form any desired pattern, such as aword, picture, symbol, animation, or other display of information. Theholes may be formed to be subtle or invisible to the eye when LED is offso that the display effectively “disappears” when it is not outputtinginformation. In this way a light pattern is transmitted through the gridof holes 304 to be visible to the user, and the surface 306 otherwiseappears blank when no light pattern is transmitted through the holes.Alternatively, other display surfaces may have identical or nearlyidentical surface quality as the arm so that the display effectively“disappears” when it is not outputting information. The display materialmay be visibly blended with the surrounding structural surface so thereis no perceivable visual break between the surface and the display. Insome examples, the material may blend or hide a material transition in away that two materials (e.g., finished metal and plastic) have avisually identical surface character. Thus, in various ways the entirearm or other system component or portion itself appears to be thedisplay in order to more effectively communicate information to the userwithin the overall human-engineered appearance of the system.

As shown in FIG. 3B, in some examples, an integral display 302 may beformed in or of a different piece of material 308 and integrated intothe arm 204 to be continuous with the arm's surface. The piece ofmaterial may be integrated in a way such that the panel of the display302 is visibly distinct from the piece in which it is located (e.g., adifferent color or texture, or a clear dividing line between structuralsurface and display), or subtly different from the piece in which it islocated (e.g., the same color, but with a barely visible join line). Insome examples, the integral display 302 may be visibly defined. Forexamples, the display 302 may include a black display area on a whitearm (see, e.g., FIGS. 4A-AB).

An integrated panel may be plastic, metal, ceramic, or other material.As described above, a grid of holes 304 may be formed in the material308. Alternatively or additionally, the material 308 may be translucent(light-conducting) or transparent so that it can conduct light fromlight source (e.g., LED) behind a plastic panel to present informationto a viewer of the panel. In some examples, an integrated panel may beformed from light-conducting Corian® material from Dupont™. In someexamples, the integrated panel may be transparent, similar to recenttelevision display technology development by Panasonic Corporation(e.g., transparent OLED display).

The integral display 302 (and any other displays) may be configured tobe readable through a transparent sterile drape that covers the link inwhich the display is mounted. And, an integral disappearing or paneldisplay may be placed on an area that protrudes from or is recessed intothe surface of a system component so that the protruding or recessedarea is visually prominent regardless of information being displayed.The protruding or recessed area may form part of the overall visualhuman-engineered design and extend beyond the boundary of the integraldisplay feature.

Controllable Displays

In various examples, a display on an arm or other aspect of a system maybe controlled (e.g., by a direct input or via another system or device(system 100, system 200, user control system 150, processing system 180,etc.) to present information such as words, symbols, or other indicia tofacilitate user interaction or understanding during a procedure. Acontrollable display may, for example, be an integral display 302 asdescribed above.

Animations may be implemented on a controllable display, which may bethe integral display 302 (e.g., disappearing display or integratedpanel), by controlling light behavior. For example, an animation maydisplay linear or curvilinear motion, such as an arrow moving in apointing direction. An animation may display angular motion, such asindicating a direction to change a kinematic pair angle (e.g., to move ajoint or adjust the position of a link or elbow with respect to anadjacent link or elbow). An animation may display size change, such asan expanding or contracting circle or disk. An animation may includecolor changes, such as changing between red and yellow, or red, yellow,and green, and such color changes may be dynamically linked to a systemor component state, such as gradually changing color as a particularlink pose is approached. An animation may include text, such asscrolling horizontally or vertically, or zooming in or out, or graduallyappearing or fading away. An animation may include morphing, such aschanging from one shape or icon to another, changing from text to ashape or icon, changing from a shape or icon to text. An animation mayinclude a combination of any two or more of the animation featureslisted above. And persons skilled in the art of animation userinterfaces will be familiar with various other animation forms that maybe used.

An animation presented on a controllable display is designed to berelevant to the clinical operation of a manipulator system or amanipulator system component. The controllable displays that display theanimations are located where they can be easily seen by a clinician whois carrying out the relevant clinical operation. As an example, a linearor curvilinear motion animation display may indicate a suggested,optional, or required direction of movement for a movable system link orother component. As another example, an animation color change may bedynamically linked to a system or component state, such as a graduallychanging color as a particular link pose is approached (e.g., centeredin a joint ROM, or approaching a joint ROM limit). As another example, amanipulator system arm has two, three, or more controllable displayslocated at various combinations of arm links or joints, and animationson the controllable displays indicate to a user how best to position thearm links by displaying a first animated movement indicator in one color(e.g., red) when a component is not in a desired position ororientation, then displaying the animated movement indicator in a secondcolor or color hue (e.g., yellow or green; lighter red hue) as thecomponent approaches the desired position or orientation, and thendisplaying either a second animated indicator or a static indicator(optionally morphed from the first animated indicator) in the secondcolor or color hue, or in a third color or color hue, when the componentis at the desired position or orientation. As another example, such ananimation scheme may be used to guide a clinical user to position themanipulator system from a first predefined pose (e.g., the IV stand poseor the transport pose) to a second predefined pose (e.g., the ready todrape pose or the ready to dock pose). As another example, an animateddisplay at a location on the manipulator system indicates that acomponent should be coupled to or decoupled from the system at thatlocation, and the animated display changes to indicate the component'scoupled or decoupled state. As another example, an animated display thatindicates the component's coupled or decoupled state further indicatesthe state of the connection—i.e., not simply physically connected butfunctionally connected as indicated by a sensor or self-test featureassociated with the connection.

The information presented on a controllable display may further indicatemany possible states, commands, or suggestions, such as an arm pose(e.g., IV stand, transport, ready to dock), or system operating modesthat correspond to the arm pose. For example, when the arm is posed fordraping in drape mode, the integral display may indicate “DRAPE” asshown in FIG. 3C., and the “DRAPE” indication disappears when the armpose is moved out of the ready to drape pose. The display may similarlyoutput other indications such as “IV STAND” or “TRANSPORT” or “DOCK”when the arm is placed in the corresponding poses as described above.

In various examples, a controllable display such as the integral display302 may be controlled to output any of the following to a clinical user,in various ways such as by an indicator, text, or abbreviation, orcombination. These display examples are illustrative and are notlimiting; persons of skill will understand that various other types ofclinically relevant information, especially information relevant to aparticular manipulator arm or to an instrument coupled to the particularmanipulator arm, may be displayed on the particular manipulator arm asan aid to clinical personnel before, during, and after surgery. And, theinformation may be displayed in any of the various ways describedherein. As examples (i) the system is in setup mode and is ready to bemoved toward docking with a cannula; (ii) the system is in standby mode;(iii) the system is ready for transport, such as with the arm stowed ina compact pose to allow system to be easily rolled on floor (e.g., IVstand pose), or the arm is posed to be easy to handle for transport(e.g., transport pose); (iv) the system is ready to be draped; (v) thesystem is properly draped; (vi) the system is ready for roll-up next toa patient on an operating table; (vii) the system is ready to dock tocannula; (viii) the system is docking with cannula; (ix) the system isdocked to cannula (x) the system is operating in a therapeutic ordiagnostic mode, such as when master/slave following relationship isestablished between master control device and instrument; (xi) thesystem is ready to change instrument mounted on the system (or in amulti-arm system indicate which arm has the instrument to be changed);(xii) an instrument is transitioning from one operating mode to anotheroperating mode, such as stapler or energy instrument firing; or (xiii)the system is approaching or is at a center of gravity limit and is tooclose to a possible tipping.

In further examples, the system may control the integral display 302 to(i) output an instruction (e.g., “DRAPE THIS WAY→”); (ii) display asystem status (e.g., “GETTING READY TO DOCK”); (iii) report a status ofarm unit (e.g., “READY FOR TRANSPORT”); (iv) show an elapsed duration ofsurgery or elapsed time under anesthesia; (v) show duration that avessel has been clamped (e.g., ischemia duration during partialnephrectomy); (vi) show countdown from pre-established time limit (e.g.,countdown stopwatch feature); (vii) time to fire (e.g., energyinstruments preparing to release energy; or (viii) show informationrelevant to an instrument mounted on the corresponding arm (e.g., numberof allowable uses remaining for the instrument, number of clipsremaining for multi-fire clip applier instrument, etc.).

In still further examples, a controllable display (e.g., integraldisplay) may be controlled to output a communication (e.g., instructionor alert or comment) between clinicians, e.g., from a surgeon to a nurseworking in sterile field, or a computer-generated communication to aclinician. For instance, a surgeon at a remote user console may indicatethat a particular instrument is to be removed, and a display on amanipulator arm corresponding to the particular instrument indicatesthat the particular instrument is in fact mounted to the arm so that thebedside clinician does not mistakenly remove the incorrect instrument.As another example, an image processing system detects an endoscopicimage degradation and the system correspondingly indicates on the arm towhich the endoscope is mounted that the endoscope should be removed andits lens cleaned.

In some examples, the controllable display may output the instrumenttype that is coupled to the arm (e.g., camera, stapler, cutter,grasper), or a status of an instrument coupled to the arm (e.g., usablelives or time remaining, status of staple cartridge, status ofelectrosurgical energy application, warning about instrument capability,etc.). Or, the controllable display may output an indication of acurrent system operating mode (e.g., following mode, commanded clutchmode, break-away clutch mode, standby mode, transport mode, or othermodes). For example, the controllable display may output information ingreen or blue during the following mode, output a flashing clutchindication in yellow or red when a clutch is commanded by pressing abutton or by “breaking away” from a commanded pose when moved by hand,and output an operating mode indication in blue or purple during thestandby or transport modes.

In various examples, the controllable display (e.g., integral display302) may be controlled to display textual forms such as words,abbreviations, or letters in either static or animated ways, or it maybe controlled to display graphical forms such as symbols (e.g., arrowsto indicate direction, a numeral to indicate a particular arm) or icons(e.g., to present a flashing lightning symbol when electrosurgicalenergy is applied).

In some examples, the controllable display (e.g., integral display 302)may be controlled to indicate a safe or correct hand contact point on anarm (e.g., “GRAB ARM HERE” or “PUSH SILVER HANDLE→”). For example, asafe or correct grab point may be indicated on an integral display whena manual adjustment of the arm is required or recommended, such aslowering a portion of the arm 204 during drape mode, or such asdirecting a user to a handle or push point during transport mode. Insome examples, the integral display may be controlled to indicate ajoint that is manually adjustable, e.g., when one or more of the armjoints are locked and one or more of the arm joints are manuallyadjustable (e.g., rotational arm joints are locked in the ready to dockmode, but prismatic vertical column joint is manually adjustable). Insome examples, if an arm link moves as the result of a user touching aforce sensor on the link, the integral display may be controlled toindicate which force sensor to actuate and how to interact with arm. Forexample, the display may present a message with a location or directionindicator (e.g., “PUSH HERE” or “PULL THIS WAY→”).

In some examples, the controllable display (e.g., integral display 302)may present an instruction, alert, or warning about manipulating orinteracting with the arm or system, such as “DON'T TOUCH THE ARM!” if aclinician touches the arm while the surgeon is operating themanipulating system in the following mode.

In some examples, the controllable display (e.g., integral display 302)may be controlled to indicate detected touch on arm or instrument, e.g.,the integral display may present a message stating “DETECTED TOUCH ATSTERILE LOCATION” or “DETECTED TOUCH AT NON-STERILE LOCATION” or asymbolic indication thereof.

In some examples, the controllable display (e.g., integral display 302)may be controlled to present an assignable or computer-determinedlabeling or description of one or more portions of the system (e.g., anarm). A label may be arbitrary, such as “ARM 1” or “ARM 2”. Or, a labelmay be based upon a function of the arm or the tool mounted on the arm,such as “GRASPER ARM”, “NEEDLE DRIVER ARM” “STAPLER ARM” or “RETRACTORARM”. The controllable (e.g., assignable) arm number/designation maysupport modularity, because two or more similar manipulator systems(modular units) may be used together as described herein, and confusionamongst the systems may be inhibited or reduced by the labeled number ordesignation on the integral display (e.g., Arm 1 can be differentiatedfrom Arm 2 by using the displayed labels on the arms). In some examples,integral displays on systems or arms may communicate regarding three ormore arms working together (e.g., to support two instruments and acamera, each mounted on a separate manipulator system's arm). Todifferentiate the arms, the integral displays may each display somethingdifferent (e.g. “ARM 1”, “ARM 2”. “ARM 3”).

A controllable display (e.g., integral display 302) may provide anoperator with an adaptive way to assign and reassign designations. Forexample, in a modular system, it may not be known which system will beArm 1, Arm 2, and Arm 3 (Or “grasper arm,” “stapler arm,” “camera arm”)before a procedure starts, and the systems may be assigned labels at thebeginning of a procedure, or reassigned labels during a procedure (e.g.,a label may be reassigned associated with a tool change, e.g., a “NEEDLEDRIVER ARM” label may become a “STAPLER ARM” label). The controllabledisplays may enable dynamic switching of arm designation numbers duringa surgical procedure (e.g., repositioning or reassigning arms so thatoriginally labelled “ARM 1” becomes “ARM 2” and vice-versa). In someexamples, indicia such as color, pattern, or brightness on other lightfeatures (e.g., display, rings) may match an aspect of the integraldisplay (e.g., color or arm number or name) to help distinguish portionsof the system or arms. For example, displays or light features on onemay arm appear in one color, and displays on a second arm may appear ina different color.

In some examples, the content displayed on a controllable display may betied to a step in the procedure. For example, the controllable displaymay state “staple reload required”, “move me”, or “replace my instrumentwith a stapler.” The displayed information may be triggered by an event,a system state (e.g., nearing end of range of motion or stapler empty)or may be triggered by a request (e.g., a clinician request). As anotherexample, a manipulator system detects that an arm has been properlydraped with a sterile drape, and one or more light features are lit in acolor (typically blue) to indicate the draped portion of the arm issterile.

In some examples, when an operator is in a certain step in apre-surgical or surgical procedure, the display may indicate which armto change, so that it is clear which one of the arms is the one thatneeds to be serviced. The indication may be literal (“move me”) orsymbolic (e.g., a picture of an empty stapler) or arbitrary (e.g.,flashing lights to indicate an arm to be moved or otherwise serviced. Insome examples, light features (e.g., rings) may light up or change(e.g., change color or brightness or animation state). Such a displaymay be triggered by a system event or user request. In some examples,when surgeon requests an instrument change, one or more displays on thearm may light up to indicate that it is the arm for which the change isrequested. For example, a display on an arm may output “CHANGE MYINSTRUMENT” so it is clear which one of the arms is the one that needsto be serviced with an instrument change. In some examples, thesurgeon's request may specify the instrument type to be changed, thesystem memory has stored which instrument is assigned to (mounted on)each arm, and accordingly the system may identify the appropriate arm.

In some examples, a general system or system component fault, or a jointat or near a range of motion limit, may trigger a display, such as anilluminated component or other indicia on a display screen or otherlight feature.

In some examples, information (e.g., text or graphic) on a controllabledisplay may help guide the user through steps to change a kinematicpose. For example, a display may present a static or animated “PUSHHERE→” or “PRESS CLUTCH BUTTON” or “MOVE THIS ARM”, or a similar staticor animated graphic, to indicate which structure to move or direction ofmovement.

Behavior of Light Features

In various examples, the behavior of light features (e.g., integrateddisplays, lighted rings (see discussion below), or other light featuresmay be controlled to communicate to a user (e.g., clinician oroperator).

The light feature outputs may be various individual colors, or the lightfeature outputs may change colors, or the light feature outputs may besimultaneous multiple colors (e.g., a red-blue-red-blue pattern acrossan area). The light color may change, for example, from red to blue, orfrom red to blue and back to red, or red to blue to white to blue to red(e.g., to indicate a state or a value in a continuum or range). In someexamples, the lights may pulse or flash (e.g., pulsed LED or flashingLED). A pulse may be a discrete pulse or flash, or a soft transition(e.g., a “breathing” light having a pulse frequency of 1 second, 2seconds, 3 seconds, or 5 seconds, or longer). In some examples, a systemmay use a waveform to control lights. For example, a system may use asawtooth waveform pattern where a light becomes gradually brighter, andthen quickly dims. In another example, a system may use a smoothoscillating pattern (e.g., sine wave), which may cause a gradual,repeated increase and decrease of light intensity over time. Otherwaveforms may be used for brightness and color changes, and differentwaveforms or waveform frequencies may also be used to communicationsystem information to the user (e.g., a light feature outputs a smoothand slow light change to indicate a normal system state, and the lightfeature outputs a harsh or rapid light change to indicate an alert orwarning system state).

Various types of information may be communicated by using a lightfeature as described herein. In some examples, the appearance (e.g.,color, pattern, brightness, animated motion, or other visualcharacteristic) of one or more light features may correlate to systemoperating mode, arm function, or instrument function. In some examples,a light feature may communicate a status or aspect of a patient. Forexample, a light feature may track with a patient's detected heart beator respiration or it may indicate a patient health warning state.Various other examples are provided below in a section titled“Communicating Using a Light Feature.”

Patient Detection

In some examples, a manipulator system may be configured to communicatebased upon patient detection. For example, a system may sense a patientlocation and position in 3D space (e.g., using depth mapping technology,proximity sensing technology), or it may receive such information fromanother system, and the system may use the patient information todynamically configure the arm in a way that is appropriate or safe. Theconfiguration may be context-sensitive (e.g., a manipulator system orarm configuration may be based on both information about the patient andinformation about a procedure or the availability or status ofequipment, tools, or other systems). For example, the system may move toa “ready to dock” pose that will may provide adequate or safearm-to-patient clearance during surgery (e.g., an arm may be oriented ata height that is higher than a patient to avoid a collision between thearm and a patient or to orient and position an instrument mount at anappropriate height above a patient (or patient surgical site) tofacilitate a surgical procedure. In another example, a system may movean arm to a “ready for teleoperation” pose that is calculated to provideadequate or safe arm-to-patient clearance, or adequate or safearm-to-second arm clearance during surgery. In some examples, the systemmay indicate that an arm, or a specific location on an arm, is at a “tooclose to patient” pose or a “near to colliding with patient” pose. Insome examples, a system may indicate that an arm or instrument is nearor at the end of a range of motion with respect to the patient (“unableto move farther in this direction”), such as a maximum possibleinstrument pitch or yaw.

Other Display Features

The system may also control the integral display 302 to show a guidedsetup instruction set or checklist. Information on guided setup isfound, e.g., in U.S. Patent Application Publication No. US 2017/0172674A1 (filed Sep. 9, 2016), which is incorporated herein by reference.

Any of the controllable display features described above, as well as anyadditional display or visual indication features described below, may beused in a surgical environment in which the display is covered by atransparent sterile drape. A drape may be secured near or over thedisplay to make the display more visible through the drape (e.g., thedrape may be positioned flat across the display). In various examples,ties, loop and pile fasteners, magnetic fasteners, etc. may be used tosecure the drape in this manner.

Displays and Arm Operations

FIGS. 4A are 4B are illustrations of a portion of manipulating system400 that has a display 402 (e.g., organic light-emitting diode (OLED)display) on a portion of the arm 404. In some examples, the display 402(e.g., OLED display or similar display) may be located between two majorstructural portions 410, 412 of an arm 404, such as between two “elbow”links or between an elbow and a bend portion of an arm link as describedabove. For example, the display 402 may be between the first elbow 218and second elbow 220 of the system 200 shown in FIG. 2A, or between thesecond elbow 220 and the first bend 222, or between the second bend 224of the first link 216 and the third elbow 228. In other examples, thedisplay 402 may be on a major structural link, or the display 402 may beon a small separate link between two major structural links.

As shown in FIG. 4A, a user interface 403 on the display 402 may becontrolled to show user-selectable elements (e.g., virtual buttons) 414,416, 418, which may correspond to different poses of the arm 404 (e.g.,selecting a button 416 may cause the arm 404 to move into drape pose andselecting icon 418 may cause the arm 404 to move into transport pose).

As shown in FIG. 4B, the display 402 may be on an arm 404 that hasbuttons 406, 408 on portions of the arm 404 near the display 402. Theuser interface 403 on the display 402 may have one or more indicators420, 422 that points to a button 406, 408, which may for exampleindicate to a user to press the button to activate an operation of anarm 404. The user interface on the display 402 may also include anindicator 424 (e.g., a stripe) that changes (e.g., appears to scroll ormove, or changes color, size, or brightness or other appearance) as ajoint (e.g., a joint at or near where the display 402 is mounted) movesthrough a range of motion. For example, the indicator 424 may beanimated to indicate movement or impending movement, or all or a portionof the indicator 424 may change appearance (e.g., color) to indicatethat an end of a range of motion is near (e.g., change from green toyellow to red as a joint approaches a range of motion limit). The userinterface 403 may include a status indicator 426 (e.g., an icon, or textthat says “READY TO DOCK”).

In some examples, the display 402 may be configured or controlled tochange as the arm 404 or an elbow or other joint moves. The display 402may, for example, be configured to “follow the user,” so that as the arm404 or elbow moves, the information on the display 402 remains visibleto the user (e.g., see upward-angled display in FIG. 4A). For example,the display 402 may be configured or controlled so that for a pluralityof orientations (e.g., any orientation) of adjacent components (e.g.,elbows adjacent the screen) or for a plurality of poses, the display 402is at a constant orientation with respect to vertical or a specifiedframe of reference (e.g., user location, world reference frame withgravity vector defining the z-axis, etc.) so that a clinical userviewing the display 402 can see it at the same viewing angle as the poseof arm 404 changes. Likewise, control inputs such as buttons associatedwith a touch display may change orientation with reference to the arm404 as the arm pose is changed. The control inputs may remain orientedat a constant angle to a reference frame outside the system as the armpose changes. Or, as the arm pose changes the control inputs maytranslate along the display 402 so that they remain oriented towards apoint in space (e.g., when the display is relatively low with referenceto an average user, the control inputs are oriented upward, and when thedisplay is relatively high with reference to the average user, thecontrol inputs are oriented downward). In this way a clinical user caneasily view the display 402 and operate the control inputs at variousarm poses.

In some examples, a person viewing the display may select the displayorientation by translating it along the display area (e.g., by using avirtual handle on the display, or by using a touch-sensitive element,such as a touch-sensitive OLED screen), and the system may then maintainthe selected orientation with reference to a reference frame defined onor outside the system. The system may assume a viewing distance (e.g.one meter) or it may calculate a viewing distance (e.g., based onsensors that determine the position of a person relative to thedisplay). An accelerometer or an inertial measurement unit (IMU) or liketechnology may be used to define orientation of the display, the arm, orboth.

In various examples, the full circumference of an arm link or elbow maybe an OLED display (i.e., the OLED may extend all of the way around thesurface of the arm), or the OLED display may extend part way around(e.g., one quarter, or one third, or one half way around the arm).

The OLED display may be configured to present information to a user, toreceive an input from a user, or both. For example, one or more optionaltouch buttons on the display may enable a user to select one or more ofvarious system functions or modes. For example, one or more indicatorson the display may communicate any of the information described above orbelow (e.g., current pose, alerts or warnings, range of motion status,arm label (e.g., ARM 1), or connected tool (e.g., stapler)).Conveniently, the display may be positioned at or near the arm locationthat in the transport mode extends beyond the vertical boundary of thebase as a push point for the user (see e.g., FIG. 2J), or at a similarlocation when the arm is being set up for or used during surgery (seee.g., FIGS. 2E and 2I).

In some examples, one or more of the touch buttons 406, 408 on thesystem (e.g., at an elbow) may include or be combined with an indicator(e.g., light feature) on the button. For example, an appearance of oneor both of the touch buttons 406, 408 may change based on a user touchor a system event. In some examples, any of the various display outputappearances described above may be applied to the touch buttons 406, 408(e.g., a light feature on or around the button may change to indicate anidentity of the arm, a pose, a status, etc.).

Interaction of Screens and Other System Input Devices

In various examples, a display (e.g., integrated display or OLEDdisplay) may present options that are selectable with physical ordisplay buttons placed nearby, e.g. buttons 406, 408 on the elbowsadjacent to the screen. In some examples, the display 402 may indicatelocation of relevant button for a particular desired feature (e.g.,indicated using an arrow). For example, selectable options may bepresented on the display 402 and optionally indicated as associated witha particular button (e.g., using an arrow), and an option may beselected by pressing an appropriate one of the buttons 406, 408. Invarious examples, receipt of an input (e.g., button press) through abutton may move the display through screens, navigate menus, or initiatean action. In some examples one or more of the buttons 406, 408 mayoperate as a joint lock control (typically called a “clutch” feature).For example, the button may lock or unlock one or more of the joints inthe system shown in FIGS. 2A and 2B.

In some examples, a system may cause a joint or multiple joints in anarm to freeze (controllably lock) in position. In some examples, abutton push (e.g., an individual button push; a subsequent push of thesame button; etc.) may unlock a joint or multiple joints. In someexamples, joint locking and unlocking may be controlled with twoseparate buttons, e.g., one button (e.g., button 406, optionallyindicated with an indicia such as a red light feature) may be used tofor locking one or more joints, and a second button (e.g., button 408,optionally indicated with a second indicia such as a green color, whichoptionally only appears when the one or more joints are locked) may beused for unlocking one or more joints (e.g., the joints unlock andreturn to an assisted move state, a gravity compensated state, or ateleoperated state after being unlocked). In some examples, a buttonlock command may prevent teleoperated movement or movement in responseto a processing system 180 or movement responsive to user control system150. In some examples, commands from a user control system 150 orprocessing system 180 may be executed by moving other joints or aspectsof the manipulating system while maintaining a locked joint in a lockstate (e.g., to avoid a collision with an object of which the system 150or 180 may not be aware).

In some examples, a single button that may control two bi-stable jointlock/unlock modes (e.g., a first press locks one or more joints, and asecond press unlocks the one or more joints). In some examples, a codeor pattern may be required to lock or unlock one or more joints to avoidunintended commands, e.g., two buttons presses within a short time (a“double click”), or three presses in sequence within a time, or a longpress, or a short press followed by a long press, or variations thereofmay be required to initiate a lock operation, and the same or differentcode or pattern may be required to unlock the one or more joints.

Touch Points

In various examples, a system may include one or more designated touchpoints (e.g., touch sensor) at one or more locations on arm to selectvarious system functions or system operating modes. For example, thebuttons 406, 408 shown in FIG. 4B may be touch points rather thanmechanical pushbuttons. FIG. 5A is an illustration of a portion of amanipulating system 500 and a touch point 502 on an arm 504 of themanipulating system 500. FIG. 5A also shows an arm handle 506, which mayalso be a touch point.

The manipulating system 500 may initiate an action in response to touch.In some examples, a system may require a touch at a certain location tocarry out a certain action, and for such an action the touch point islocated to provide an ergonomic control for the user. That is, the touchpoint is located at or near a safe and recommended location at which theuser would or should touch the manipulating system in order tophysically carry out an action, or to observe an automated actioncarried out as the result of the touch. A touch point may be integratedinto an arm link or into a display, such as the black strip OLED display402 shown in FIGS. 4A and 4B and described above. In some examples, atouch point may be combined with any of the light features (e.g., lightrings, other controlled displays) described above. For example, a lightring, a portion of the light ring, or a lighted or unlighted portion ofa light ring (which may dynamically change) may be a touch point.Similarly, a touch point may be located near such a light ring or lightring portions (e.g., a touch sensor ring adjacent the light ring). Otherlight features may function similarly. In various examples, a lightfeature may indicate where to touch, direction of arm movement, or anarm or system status (e.g., ready to move). A light feature itself maybe the touch point that is responsive to touch, or the light feature maybe adjacent the touch point.

Handles

A manipulating system may include one or more handles, such as avertically-oriented handle (e.g., handle 506 shown in FIG. 5A, shown atan acute angle upward relative to the plane of the floor), ahorizontally-oriented handle (e.g., handle 506 shown in FIG. 5B, shownoriented generally parallel relative to the plane of the floor; handleportion 805 of the helm 213 shown in FIGS. 2A and 8 ; or handle 702 inFIGS. 7A-C), or handles in other orientations or variable orientations.The system may have a mini handle, or a mini helm, an egress handle(e.g., to trigger an egress of an instrument from a patient or a systemfrom a surgical table), an emergency handle (e.g., to trigger anemergency stop of an operation of the system), a sterile handle, or anon-sterile handle, or any combination thereof. The handle 506 shown inFIG. 5A may, for example, be the handle 227 shown in FIGS. 2A and 2B.

In some examples, the handle may have an automatic leveling feature. Forexample, a system may be configured so that during a change in arm posea handle remains level (i.e., horizontal with reference to the floor),or the handle remains at a pre-defined (e.g., +20 degrees with referenceto the floor) or user-defined (e.g., user has placed handle at acomfortable orientation for the user's height) angle, or the handlemoves to a predetermined or context-appropriate orientation so as to beaccessible throughout a range of arm positions or orientations, e.g., asthe arm moves to various kinematic poses such as moving into atransport. IV stand, or docking pose. A manipulating system may includeone or more motors that may be controlled to control an angle of ahandle with respect to an arm as determined by one or more sensors thatdirectly or indirectly sense joint angle, or as determined with respectto a gravity vector or some other reference orientation outside thesystem as determined by an associated sensing system (e.g.,accelerometer, indoor positioning system, etc.).

In some examples, a manipulating system may include a level sensor, andlevel sensor data or information based on level sensor data may be usedas an input enable leveling of the handle as the arm moves. A levelsensor may be inside a handle, or it may be inside a structure proximatea handle, such as an arm or elbow. A manipulating system may maintain ahandle at an angle other than horizontal, e.g., a sensor may determinehorizontal, and a handle may be maintained at a specified angle or rangeabove or below horizontal, e.g., 30 or 45 degrees above horizontal (seeFIG. 5A which shows a handle 506 at about 45 degrees above horizontal;FIGS. 2B, 2D, and 3A which show a handle 227 level with horizontal; FIG.5B which shows a handle 506 level with horizontal; FIGS. 7A-7C and 9A-9Dshow a handle 702 level with horizontal), and the specified angle may beconstant, or it may be determined based upon sensor input or a computedstate (e.g., the handle may rotate up or down based on height to extendtoward a user) or context (e.g., the handle may rotate into a specifiedorientation that corresponds to a pose).

In some examples, a system may include an auto-angle feature so thehandle 506 is configured at a convenient angle for the user. Forexample, FIG. 5B is an illustration of a manipulating system with ahandle 506 presented to a user. The system may move the handle 506 so itis oriented toward a user. For example, a system may control a handle sothat the handle angles upward from horizontal (e.g., as shown in FIG.5A) when the arm is in a pose in which the handle is relatively low withreference to an average user, and the system may move the handle so thatthe handle angles downward from horizontal when arm is at a pose inwhich the handle is relatively high with reference to an average user.In addition, any of the handle orientation features described above mayalso be applied to controlling orientation of a handle.

In another example, a manipulating system may maintain a stored memoryof user height or user preference for handle height or orientationangle, and as the arm changes position, orientation, or kinematic pose,the system may move the handle height or orientation based on storeduser information.

In some examples, a handle may include buttons on the inside of thehandle, for example at inside surface 510 shown in FIG. 5A. In variousexamples, one or more buttons may (i) control the handle's movement(e.g., actuating a button may control one or more motors that moves thehandle); or (ii) may release (“clutch”) the handle to enable manualadjustment of the handle, or lock the handle; or (iii) buttons maycontrol other system functions.

In some examples, a handle may include an arm or instrument egressfeature. An instrument egress feature may cause an instrument toautomatically retract from the patient. An arm egress feature may causean arm or an entire system to automatically perform actions such asundock from a cannula, move to a pose that allows a clinician to movethe system away from a surgical table, or move away from a surgicaltable. Instrument and arm egress features may optionally be combined. Insome examples, a specific handle motion (e.g., rotation through aspecified angle or a pull of the handle through a specifieddisplacement, or a combination of movements) may initiate an instrument,arm, or combined instrument and arm egress. An egress may also beinitiated by via an input through one or more buttons on the handle orthrough a display (e.g., OLED display), or through a display on ahandle. In some examples, a system may include a dedicated handle 702,as shown in FIG. 7C. Light features as described above may outputinformation associated with an instrument or arm egress function, suchas indicating the arm associated with the instrument withdrawal, orindicating which arm or arm joint is about to move and optionally inwhich direction, or indicating which manipulating system is about tomove away from the table.

In some examples, a manipulating system handle as described herein maybe configured to be sterile (e.g., in the sterile field). A sterilehandle may be draped (by a sterile drape). In some examples, a handlemay not be sterile, or a first handle may be sterile, and a secondhandle may be non-sterile. A sterile or non-sterile handle (or both) maybe indicated by an appearance associated with its function, such as acolor, pattern, or material.

Any of the handles described herein may be used for general transportwhen a system is stowed or otherwise placed in a kinematic pose suitablefor storage or transport (e.g., IV stand pose, transport pose). Forexample, the handle 506 shown in FIG. 5B may be used to assist withnavigation by providing a convenient grab point to push, pull, or steerthe arm or activate controls on the handle (e.g., to initiate amotor-assisted transport mode, which may reduce the amount of push forcerequired to move the unit).

A handle may be sized, shaped, or contoured to facilitate easy andcomfortable grabbing or holding by a user. For example, as shown in FIG.8 , a handle 804 may have a generally oval shape, with an optionalflare-out 816 (e.g., generally egg-shaped) at the grab location 814.

A handle may be formed from one or a combination of materials, e.g.materials that are both strong (to facilitate pushing and pulling) andeasily amenable to sterilization. For example, a handle may be made froma high-grade metal, such as aluminum or an aluminum alloy. A metal(e.g., aluminum alloy) handle may be polished or otherwise finished togive a clean appearance that is consistent with the human engineereddesign principles discussed above. In some systems, the handle materialand appearance may be consistent for two or more unit handles or othertouch points, so that touch points have the same appearance for ease ofuse and clarity of communication to a user (e.g., clinician). Forexample, when two or more handles have the same general appearance suchas rounded and polished aluminum, a user may then learn and understandthat rounded and polished aluminum handles, even if they have size andshape variations, provide a safe or appropriate touch point to touch,pull, push, or otherwise physically interact with a manipulating system.Further, if two manipulating systems each have different kinematicarchitectures, a consistent general appearance of handles on bothsystems visually reassures the clinician of the proper grab location oneither system. For example, one or more rounded and polished aluminumhandles may each be used on a first manipulating system that has asingle manipulator arm (see e.g., FIG. 2A) and on a second manipulatingsystem that has two or more manipulating arms (see e.g., FIG. 1B).Likewise, one or more rounded and polished aluminum handles may each beused on a first manipulating system that has a manipulating arm havingone kinematic architecture (see e.g., FIG. 2A) and on a secondmanipulating system that has a manipulating arm having a secondkinematic architecture different from the first kinematic architecture(see e.g., FIG. 1B).

In some examples, a handle may align with a manipulating system controlhelm, such as the helm 213 shown in FIGS., 2A, 2B, and 8. Initial helmor handle orientation changes may be automatically controlled by thesystem or manually by a user, and the corresponding helm or handleorientation change may also be automatically controlled by the system ormanually by a user. For example, a handle may align with a control helmin a specified pose. In an implementation in which the handle andcontrol helm are independently movable, the handle may follow the helmorientation, or the helm may follow the handle orientation, or both. Forexample, if a user changes a helm orientation (e.g., angle withreference to the base) is from one orientation to a second orientation,the handle may automatically adjust to align with the second helmorientation. Similarly, if a user changes a handle orientation, the helmmay automatically adjust to align with the second helm orientation.

A handle may be located between links in a manipulating system (aninter-link handle), or a handle may be integrated into a link in amanipulating system (an intra-link handle), either at an end of a link(e.g., adjacent a joint with another link) or mid-link spaced away fromthe link's opposite ends. If a handle is moveable with reference to alink, the handle itself is a kinematic link and is coupled to the linkat a handle joint. Handle 212 (FIGS. 2A and 2B) is an illustration of anintra-link handle because it is at the top end of the column 210 thatextends from the manipulating system base. Handles 227 and 506 areillustrations of inter-link handles because they are located between twoarm links.

FIG. 6A is an illustration of a mid-link intra-link handle 602 on link604 of a manipulating system arm (e.g., FIG. 2A, arm 204). FIG. 6B is aside perspective view of the handle 602 shown in FIG. 6A. In someexamples, the handle 602 may be used as a grab point for moving a systemor arm. And, the handle 602 may include other interaction features asdescribed above, such as controllable displays, light features,mechanical buttons, etc.

As shown, handle 602 is a generally oval shape having a first end 602 aand an opposite second end 602 b. Handle 602 includes an outer surface606 oriented outward from link 604. Handle 602 also includes a sidewallsurface 610, which as shown is a continuous surface around the handle602 but optionally may be two or more discrete surfaces. One or moreoptional finger recesses 605 may be included in the sidewall surface(s)610 (e.g., one finger recess 605 on each side of the handle 602; afinger recess 605 may extend partially into or completely through thehandle 602) to allow a user to more firmly grasp the handle 602. Andhandle 602 is optionally located within a recess 603 in link 604.

As shown, the design of a mid-link intra-link handle is consistent withother inter- or intra-link handles and is consistent with the overallhuman-engineered appearance of a manipulating system, and as a resultsafety and approachability are conveyed to a clinical user, and also theuser's eyes are drawn to the mid-link handle as an interaction locationon the system. The mid-link handle may be visually configured to providea consistent appearance with one or more other inter- or intra-linkhandles on a manipulating system so as to indicate it is an acceptabletouch-point (e.g., safe to grab or designed for touch or grasping). Forexample, inter-link handles 212 and 506 as described above have arounded shape and a polished aluminum surface finish, and so a similarappearance is given to mid-link intra-link handle 602 by rounding itsends 602 a. 602 b and providing a polished aluminum surface finish tothe handle's sidewall surface 610. Optionally, the polished aluminumsurface finish is extended part way into outer surface 606 in aperimeter region 606 a so that when handle 602 is viewed straight on,(i.e., the view shown in FIG. 6A), the rounded and polished aluminumvisual appearance indicates to the clinical user that handle 602 is anintended touch point consistent with other touch point appearances onthe manipulating system. Outer surface 606, or the inner portion ofouter surface 606 bounded by perimeter region 606 a, may be the samecolor as link 604 to provide a consistent overall visual appearance tothe link 604 and the manipulating system as a whole. Further, all or aportion of outer surface 606 optionally includes a controllable displayas described above, and below with reference to handle 702 (FIGS.7A-7C). And still further, a light ring feature is optionally includedin link 604 to fully or partially surround handle 602, or is optionallyincluded in handle 602 to fully or partially surround handle 602's outersurface 606 (e.g., within perimeter region 606 a; within sidewallsurface 610 where it joins outer surface 606).

The rounded shape and the polished aluminum surface finish of thevarious handles described herein are illustrative examples of anergonomic design principle for a manipulating system, and skilled userswill understand that various geometric shapes (e.g., circular, oval,triangular, rectangular) and various surface finishes (e.g., color,surface texturing, material, light feature, etc.) may be usedconsistently on a manipulating system (e.g., two, three, four, or morehandles as described herein) to designate intended and safe touch pointsto a clinical user. In addition, a visual border may optionally beplaced around a handle to highlight the handle's visual characteristicsto a user (i.e., to communicate that a handle is at this location). Forexample, with reference to FIGS. 6A and 6B, link 604 and handle outersurface 606 may be white, and recess 603 may be dark grey to highlighthandle 602's polished aluminum surface finish as consistent with asimilar polished aluminum surface finish on handles 212, 227 and/or 506.

As shown in FIG. 6B, handle 602 may be configured to move with referenceto link 604 from a stored location outward (as indicated by the arrow611) to a location in which it is protruding from link 604's surface.Outer surface 606 of the handle 602 may be a continuation of the outersurface contour of link 604 so that in the stowed position, the contourof outer surface 606 is continuous with the outer surface contour oflink 604, which preserves the overall contour of the link 604 and thelink's human-engineered features as described above (e.g., safe andapproachable; constant cross-sectional area along the length of thelink, etc.). For example, an outer surface 608 of the link 604 may havea cylindrical shape as described above, and the outer surface 606 of thehandle 602 may be a continuation of the cylindrical outer surface 608 ofthe link 604. As described above, handle 602 may be within a recess 603when in the stowed position, and so an outer portion of the handle 602is exposed when the handle 602 is stowed. When the handle 602 is in theprotruding position, recess 603 provides space for the user to grab thehandle 602 or touch one or more interaction features in the handle 602,and so the handle 602 is required to move a relatively shorter distanceinto the protruding position for effective user access. In somealternate examples, there is no recess 603, and the outer surface 606 ofthe handle 602 may be flush with outer surface 608 of the arm. In thissituation, the handle 602 is required to move a relatively longerdistance into the protruding position for effective user access. In oneexample, either with or without optional recess 603, optional fingerrecesses 605 are hidden when handle 602 is in the stowed position andexposed when handle 602 is in the protruding position. The protrudingdistance shown in FIG. 6B is an example, and longer or shorterprotruding distances may be used.

Handle movement from the stored position to the protruding position, andfrom the protruding position to the stored position, may be controlledby user command, by a manipulating system controller in response to astate or event, or a combination of both user and manipulating systemcontrol. In some examples, a user touch point (e.g., force sensor, OLEDtouch screen area, finger swipe detector, mechanical button, etc.)either on or near the handle receives a user command to extend thehandle, and then manipulating system carries out the received command.In some examples, the manipulating system detects a user's hand presenceon or near the stored handle (e.g., touch sensor, force sensor, motionsensor, etc.) and in response extends the handle. In some examples themanipulating system extends the stored handle when a particularoperating mode is entered because the extended handle is associated withthe particular operating mode, such the handle extending when the readyto dock mode or the IV stand mode as described above is entered. In someexamples a system event will cause the system to extend, such as when asystem fault is detected or if general system power failure is detected.A handle default position may be either protruding or stored.

Methods used to retract the handle from the protruding position to thestored position may be similar to the methods used to extend the handleas described above. In addition, in some examples, the handle 602 mayautomatically retract from the protruding position to the stowedposition after the handle is used. For example, the handle 602 mayretract after sensing that has not been used. For example, the handlemay include (or be coupled to, or controlled based upon) one or moreuser presence sensors, such as a force sensor, capacitive sensor (todetect touch), or motion sensor (e.g., to detect motion near thehandle). The sensor input may be used to determine (e.g., by a controlsystem or a handle control circuit) whether the handle is still beingused, or whether a user or user's hand is still near the handle, and thehandle may be retracted after it is determined that the handle is nolonger being used or grasped. In some examples, the handle 602 mayretract after expiration of a delay period (e.g., after a few seconds).For example, the handle 602 may retract after sensing that a touch orproximate object (e.g., a hand) has been removed or that no touch hasoccurred for the delay period (e.g., no interaction for a few seconds).In some examples, the handle 602 may retract in response to touch (e.g.,a force over a threshold limit toward the stored position) may cause thehandle to move toward the stored position). For handle retraction,either by manual command or automatically, an optional safety featuremay be included that detects a resistance to the handle retraction(e.g., an excess motor current is detected), and handle retractioneither stops or reverses so as to not trap the user's hand in theretracting handle (e.g., in the finger recesses). In some examples, thisoptional safety feature is not included if handle design is such thatthere is no possibility of user injury during retraction (e.g., sidewallsurfaces are smooth and cannot trap a user's finger when the handleretracts).

Aspects of handle movement and movement control associated with handle602 have been described in terms of translational motion, and theseaspects also apply to handle movement and movement control associatedwith rotational motion of other handles described herein. In a waysimilar to the stored and protruding handle 602 positions describedabove, handles such as handles 212, 227, 506, and 702 may rotate from astored orientation in which the handle does not protrude from themanipulating system or arm to an extended orientation in which thehandle may be grasped or interacted with by the user. For example,handle 227 shown in FIG. 2B in a protruding orientation may optionallyrotate to a stored orientation that is not accessible by the user (e.g.,parallel to and behind link 214 as illustrated in FIG. 2G or 2I). Thus ahandle may have two or more position or orientation locations dependingon various user inputs and conditions.

With reference to handles in general as described herein, in someexamples an orientation or position location (or default orientation orposition location) of a handle (e.g., handle 602 or any other handledescribed or shown herein) may vary based on a manipulating systemcondition or event, such as an operating mode or arm posture. Forexample, a manipulating system may extend or rotate a handle based on orresponsive to satisfaction of a condition, e.g., the system may extendor rotate a handle for or use in an operating mode in which it may beneeded (e.g., responsive to a transition of a manipulating system to atransport or an IV stand pose), or a manipulating system may retract orrotate a handle responsive to satisfaction of a condition (e.g., ahandle may retract or rotate responsive to docking of a cannula to thesystem). As shown in FIG. 2G, handle 602 may be useful for moving amodular system when the system is in an IV stand-like pose or anotherpose.

In some examples, one or more “handle out” states (in which the handleis extended or rotated to a protruding position, or it is maintained inan extended protruding position), may be predefined. A handle out statemay be declared or a handle may be extended (or maintained in anextended position) in response to satisfaction of a condition. Forexample, a handle may be extended or rotated during transport mode orready to dock mode. In some examples, a “handle out” state may occur(and the handle may be extended) when a system senses that a handle maybe needed, such as when a joint or link reaches a range of motion limitthat would require the user to manipulate the joint or link. In someexamples, a handle may extend or rotate when system switches totransport mode, or when the system determines that the arm is posed inaway that the handle may be useful for transport.

In some examples, a “handle in” stored state may be declared or a systemmay retract or rotate a handle (or maintain a handle in a retractedposition) based on or responsive to a condition, e.g., when the systemis in an operating mode in which the handle should not be used. A“handle in” state may be predefined. For example, a handle may beretracted or rotated, or it may stay retracted, while a clinician isoperating a system in a “following” mode (e.g., a mode where the systemfollows movements or inputs from a user control system), or when asystem enters cannula docking mode (e.g., after an arm is positioned ororiented to place its associated instrument mount over a cannulainserted into a patient, at which point the handle may no longer beneeded), or when a system determines (e.g., senses or receives an input)that an arm is docked to cannula, or when a system determines (e.g.,senses or receives an input) that an instrument is mounted on the arm.In some examples, a “handle in” state may be declared when the systemsenses the arm is in a pose in which the handle should not be availableor should be retracted. For example, when a system senses that a handleis at a potentially unstable leverage point that may cause result inunintended movement due to stability (or instability) around a base, aclutch status, or a movability (or immovability) of one or more arms orjoints.

The handle 602 (or any other handle described herein) may include one ormore force sensors, which may provide input for moving an arm or systemsuch as the system 200 or arm 204 shown in FIGS. 2A and 2B. For example,a manipulating system may provide power-assisted motion (e.g., bycontrolling motors that move joints or system links or other components)based on force or torque sensed at a handle, and so a small forceexerted by the user at a handle location is sufficient to easily movethe system, arm, or component that would otherwise require a relativelylarge force.

FIG. 7A is an elevation view illustration of a handle 702. FIG. 7B is aperspective illustration of the handle 702 shown in FIG. 7A. FIG. 7C isa plan view of the handle 702 shown in FIGS. 7A and 7B. The handle 702may include an integrated display 706, which may for example be a userinterface screen (e.g., touch screen) or other controlled display asdescribed above. The handle 702 may include a first portion 708 that mayinclude the integrated display 706, and one or more side portions 710,712 that may connect the first portion 708 to an arm 704. The handle 702may be both a handle and a display. Any of the features described abovewith respect to handles or displays (e.g., control of orientation orinformation on the display, release or locking of movement) may also beapplied to handle 702 and display 706.

As shown in FIG. 7A, the display 706 may include a user interface 720,which may include one or more indicators or user interface elements(e.g., virtual buttons). An indicator or user interface element may, forexample, include text or a symbol, and may be controlled to present acolor, brightness, pattern, or other appearances to communicateinformation to the user. For example, the user interface 720 may includea status indicator 722, which may for example include text that says“READY” or “SYSTEM IS NOW READY.” The user interface 720 may include agraphical status indicator 724, such as a check mark. For example, aninterface may present a white circle with a green checkmark in thecircle, which may indicate a status (ready) or condition (e.g., toindicate that start-up checks were completed successfully.) The userinterface 720 may include an indicator of a next step 726, such as antextual or graphical indicator to dock (e.g., “DOCK NOW”).

As shown in FIG. 7C, a handle 702 may include an emergency feature,e.g., pulling the handle as indicated by the arrow may trigger anemergency action such as (i) stopping movement of the arm or anothercomponent of the system, or (ii) stopping an actuation of a surgicalinstrument, or (iii) triggering a manipulating system egress procedurefrom the patient (e.g., withdrawing an instrument, or moving a systemaway from a surgical table or other object). The handle 702 may includea symbol that may inform a user of a direction or technique foractuating the emergency feature of the handle (e.g., the symbol mayindicate to pull the handle). In accordance with human engineeringconsiderations as described herein, handle 702 and its associated pullmotion feature may be located on a manipulating system where the pullmotion is in the direction of desired system or arm movement associatedwith the manipulating system control feature controlled by the handlepull feature. For example, in a situation in which a manipulating systemis required to egress from a patient bedside, the direction of thehandle's pull-motion feature is the same direction as required forsystem egress, and the handle location is where a user would normallypull to move the system away. And, the handle pull motion feature mayinitiate one or more automatic actions required for egress, such as asequence in which an automatic instrument withdrawal is accomplished,and then an automatic undocking from the associated instrument cannulais accomplished, and then system wheel brakes are automatically releasedto allow the system to be pulled or motored away. In this way a single,intuitive clinical user movement of pulling the system away from thebedside automatically completes all system actions necessary to allowthe system to safely disengage and move away from the patient.Optionally, handle movement or force sensors detect a clinical user'spull direction, and system egress direction corresponds to the handlepull direction. And, display features optionally output information thatinforms the user of the progress of the actions required for egress(e.g., a flashing red light feature at the distal end of the armassociated with instrument withdrawal until complete, then all lightfeatures on the arm change color change from red to green when cannularelease is complete, and then a flashing green light on the baseindicates that brake release is complete). Many other actions andsequences of actions may be controlled by a handle motion, and manyother light feature outputs associated with such automatic actions andsequences of actions may be employed. In addition, corresponding audiooutputs may reinforce the light feature outputs (e.g., “wait forinstrument withdrawal”, “wait for cannula undocking”, “OK to movesystem”, oscillating warning tone, etc.).

In some examples, the arm may include one or more joints at the handle,e.g., a first elbow portion 752 or second elbow portion 754 may rotatewith respect to the handle 702 (e.g., the handle 702 may be on a smalllink between two elbow links).

Alternatively, in some examples, the arm 704 may include a double-elbow750 that may include a first elbow portion 752 and a second elbowportion 754 that may be at a different orientation or angle from thefirst elbow portion 752 (e.g., second elbow portion 754 may beorthogonal to first elbow portion 752). The double elbow 750 may beincorporated into the system 900 shown in FIGS. 9A-C. The system 200shown in FIGS. 2B-2C may also be adapted to include a double-elbow. Thearm 704 may include a first light feature 756 (e.g., a light ring) at aninterface between the double elbow 750 and an arm component 758 (e.g.,an arm link) and a second light feature 760 (e.g., a light ring) at aninterface between the double elbow 750 and another arm component 762(e.g., a forearm).

Further Manipulating System Examples

FIG. 9A is an illustration of another example teleoperated surgicalmanipulating system 900 having a manipulator arm 904 mounted on a base902, which may be mounted on wheels 910 that are coupled to the base902. While wheels 910 are shown in FIGS. 9A-9C, the example base 902 ofFIGS. 9A-9C may alternatively be adapted with one to five or more rollerballs or casters (as described in reference to FIGS. 2B and 2C) oromniwheels instead of or in combination with wheels 910, e.g., to enablesystem movement in one or more various directions (or omnidirectionalmovement).

The arm 904 may include a vertical column 930, which may include a bendportion 931. The column 930 may be rotatable relative to the base 902.

The arm 904 may include a first elbow 934 coupled to the bend portion931 of the column 930. The arm 904 may include a first link 938, whichmay include a first bend portion 936 coupled to the first elbow 934. Thefirst link 938 may be coupled to the double-elbow 750 shown in FIG. 7B.Alternatively, the first link 938 may be coupled to single elbows. Asecond link or forearm 940 (e.g., forearm) may be coupled to thedouble-elbow 750 (which is optionally two elbow links coupled directlytogether or via a short handle link). The forearm 940 may extend to awrist 942 that may be coupled to a spar 944, which may be a telescopingspar. As described above, the forearm 940 may be thinned on an underside941 to provide additional access space near the patient.

The arm 904 may include a first light feature 912 that may extend partway or all of the way around the base 902. The light feature 912 may beat or near an interface between the base 902 and the floor so as toeffectively communicate visual information about that interface. Asecond light feature 916 may be at an interface between the base 902(e.g., a top surface 914 of the base 902 as shown) and the column 930. Athird light feature 918 may be at an interface between the bend portion931 of the column 930 (or a moveable optional elbow coupled to the topof the column 930) and the first elbow 934. A fourth light feature 920may be at an interface between the first elbow 934 and the first link938. A fifth light 922 feature may be at an interface between the firstlink 938 and the double elbow 750. A sixth light feature 924 may be atan interface between the double-elbow 750 and the forearm 940.

The arm 904 may include the handle 702 shown in FIGS. 7A-7C. And, any ofthe handles, displays, light features, control methods or systems orother systems described herein may be applied to the system 900.

The base 902 may be larger on one side 908, e.g., for weighted stabilityor extra leveraged support during reach (e.g., extension of arm 904 intocantilever from base 902), and the base 902 may be smaller on the otherside 906 for close access to table, as shown in FIG. 9B. Stated anotherway, column 930 optionally extends from a location on base 902 that isnot the center of the base 902.

FIG. 9C is an illustration of two manipulating systems 900, 901 and asurgical table 950. The manipulating systems 900, 901 each may belocated across from each other on opposite sides of surgical table 950,and arms 904, 905 may extend toward each other across the table 950. Asshown, each manipulating system arm 904, 905 may extend past the otherso that each arm has access to a cannula located past a patient'ssagittal midline. Alternatively, two or more manipulating systems may belocated on the same side of the operating table, as illustrated in FIGS.2F and 11 , each with arms that extend beyond the patient's sagittalmidline. Such arm reach allows effective access to minimally invasivesurgical sites at various locations inside patients.

FIG. 9D is an illustration of a manipulating arm portion of amanipulating system showing example rotational movement degrees offreedom for a distal arm portion. FIG. 9E is a perspective illustrationof a portion of the arm shown in FIG. 9D, showing a roll axis, yaw axis,and pitch axis. As shown, the arm may be operated with software controlto rotate a defined axis, such as an axis along which a surgicalinstrument shaft is inserted and withdrawn, at a point that is fixed inspace with reference to the base 902 or to another reference frame apartfrom the manipulating system 900. This point is typically known as acenter of motion. And, if the center of motion is located in space awayfrom the arm, then the point is typically known as a remote center ofmotion. Therefore, the arm (and likewise the manipulating arm shown inFIGS. 2A and 2B, as well as similar manipulating arms) may be operatedas a software-constrained remote center of motion manipulator arm.Alternatively, the arm may be operated without software constrainingmotion around a remote center of motion. Because of the large potentialsweep volume of such software-constrained remote center of motion arms,either when operating with a constrained remote center of motion orotherwise, the human-engineered ergonomic features as described hereinare especially advantageous for operating such arms in a surgicalenvironment.

The forearm 940 may be configured to rotate at joint 970 around rollaxis 971, which may align with a longitudinal axis of the forearm 940.The wrist 942 may include a discrete spar yaw joint 972 and a discretespar pitch joint 974. The wrist 942 may be configured to rotate at aspar yaw joint 972 around a yaw axis 973. The spar 944 may be configuredto rotate at a spar pitch joint 974 around a pitch axis 975, which asshown does not intersect yaw axis 973. The system 900 may manipulate thespar 944 (and a connected item such as a cannula or surgical instrument,not shown in FIGS. 9D-9E) using three degrees of freedom (roll, yaw, andpitch) to achieve a desired cannula orientation. For example, the rollof the forearm 940 around the forearm 940's longitudinal axis, alongwith spar yaw around spar yaw axis 973 and spar pitch around spar pitchaxis 975 at wrist 942, may permit surgical instrument movement in threedimensions and rotation around three axes. It can be seen that thearchitecture of the arm 904 includes redundant degrees of freedom sothat, for example, a cannula mounted to the arm 904 can be rolled alongthe cannula's longitudinal axis by rotating joints 972 and 974, while atthe same time rotating joint 970 and extending column 930 upward to keepthe cannula in position. In addition, the base 902 may be movable on thefloor along with the various arm joints to enable the spar 944 and itsattached instrument or cannula to be positioned at a desired pose toconduct surgery. The arm 904 may include an instrument carriage assemblythat translates along the spar 944 to insert and withdraw a surgicalinstrument mounted on the carriage assembly, as illustrated in FIG. 2D.

The various arm joints (including for example the wrist and translatinginstrument carriage joints, as well as joints between elbows and largeor small links) enable the instrument to be placed in a desiredCartesian six degree of freedom pose during teleoperation. The armjoints may, for example, enable an instrument to pitch, yaw, and rollaround remote center of motion at the patient's body wall, and allow theinstrument to sway, heave, and surge in 3D space for set up or for othersurgical operation, e.g., to manipulate the instrument to a desiredposition and orientation. The position of the base 902, pose of the arm904, rotational orientation of the forearm 940, and pitch and yaworientation of the spar 944 may all be manipulated to achieve a desiredposition and orientation of a cannula or instrument over a surgicalentry site or a desired orientation and position of an instrument at asurgical site within a patient. While the manipulation of systemcomponents has been described in reference to the system 900 shown inFIGS. 9A-E, the features and characteristics described may also beapplied to the system 200 shown in FIGS. 2A-2J or other systems withsimilar configuration options.

In the examples shown in FIGS. 9D and 9E, a spar pitch joint may beinside the spar body (or within the space generally outlined by the sparbody), and the spar yaw joint may at the distal end of the forearm link.Other configurations of the axes or joints are also possible. Forexample, a spar pitch axis and spar pitch joint may be at the end of theforearm 940, and a spar yaw axis and spar yaw joint may be inside thespar body (e.g., see FIG. 10 , elements 1005 and 1006). Optionally, aroll joint may also be incorporated at an end portion of the forearm 940or inside the spar body.

FIG. 10 is an illustration of distal portion 1000 of a manipulatingsystem. An instrument carriage on which an instrument is mounted is notshown. An instrument mount assembly 1002 may be coupled to a distalportion of an arm 1002 at a wrist 1005, which may have two (or more)degrees freedom, as described in reference to the example wrist shown inFIGS. 9D and 9E. For example, the instrument may rotate around a pitchaxis 1006 and around a yaw axis 1008. The yaw axis and yaw joint may beinside the spar body 1010 (e.g., inside a housing portion) of theinstrument mount assembly 1002. The instrument mount assembly 1002 mayinclude a spar 1012 on which an instrument carriage is mounted. Invarious examples, the distal portion 1000 shown in FIG. 10 may becombined to be a part of the system 200 shown in FIGS. 2A-2B (i.e., spar1012 may be spar 209), or part of the system 900 shown in FIGS. 9A-9C(i.e., spar 1012 may be spar 944), or part of the manipulating system100 shown in FIG. 1B (i.e., spar 1012 may be spar 124). The spar 1012may be configured to mount with a surgical instrument carriage andinstrument as illustrated in FIG. 2D (e.g., an instrument designed forthe da Vinci Xi® surgical system commercialized by Intuitive Surgical,Inc., Sunnyvale, Calif.).

Referring again to FIG. 10 , to control instrument insertion andwithdrawal, a surgical instrument may be mounted on an instrumentcarriage that is mounted to the spar 1012. The spar 1012 may telescopeto move the instrument carriage, or the carriage may move along the spar1012 (e.g., along rail 1014), or a combination of both. An instrumentsuch as a cautery tool, cutter, camera, stapler, or grasper may bemounted to the instrument carriage. The instrument carriage may betranslated along the rail 1014 to insert or withdraw an instrument withrespect to a cannula or patient, as shown and described with respect toFIG. 2D. A cannula (not shown in FIG. 10 ) may be mounted to the cannulamount 1016. The instrument carriage may be driven by lead-screw (notshown). The instrument carriage may be the distal-most link in akinematic arm of the manipulating system, and it may be teleoperated,e.g., the movement of the instrument carriage may be controlled viainput from user control system 150. The position of the spar 1012 mayalso be manipulated, as described above, which provides additional rangeof motion for an instrument during a surgical procedure. The spar body1010 may include a light feature 1018, which may, for example, indicatean instrument status or type, or a range of motion status, or otherinformation clinically relevant to the distal end of the arm, such asinstrument withdrawal from the patient.

FIG. 12 is a schematic illustration of example electrical componentsystem 1200 of an example manipulating system. These components may becoupled via one or more system wired or wireless connections 1201 thatmay one or more provide command information, data, or power to themanipulating system. Either centralized or distributed processing may beused to control various system 1200 functions.

The system 1200 may include one or more integrated displays, such as afirst integrated display 1202 and second integral display 1204, each ofwhich may be integral with an arm (not shown, see, e.g., FIGS. 2A and9A) or other component, such as integral display 302 shown in FIGS.3A-3C or display 402 (e.g., OLED display) shown in FIGS. 4A-4B. Thedisplays may be coupled to the bus or a wireless connection (or both),which may provide (one-way or two-way) command information or data orpower.

The system 1200 may also include one or more light features, such asfirst light feature 1206, which may be at an interface between a column(see, e.g., structural components of FIGS. 2A and 9A, described above)and an arm, a second light feature 1208, which may be at an interfacebetween a column and a link or elbow, and a third light feature 1210,which may be at an interface between a first link or elbow and a secondlink or elbow. Any of the light features 1206, 1208, 1210 may extendaround an interface, and may be controlled by a processor, such asprocessor 1250.

While one processor 1250 is shown, the system 1200 may include multipleprocessors, which may communicate, and may be dedicated to specificfunctions or devices, e.g., a processor may be dedicated to controllinga group of light features, or motors and light features on a particularpart of the arm. Processor 1250 may be part of a manipulating system, ormay be part of a user control system (e.g., control system 150 shown inFIGS. 1A and 1C) or part of an auxiliary system (e.g., auxiliary system175 shown in FIGS. 1A and 1D). The processor(s) 1250 may controls (e.g.,send control signals) integrated displays, light features, motors,sensor activation, modes, communication signals, or other operations.The processors(s) 1250 may receive and process sensor data (e.g., forcesensors, collision detection, clutch functions mode). In some examples,the processor(s) 1250 may also perform other functions (e.g., instrumentcontrol or vision).

The system 1200 may include one or more communication circuits 1212,which may for example be or include an antenna or protocol circuit orboth. The communication circuit may, for example, be configured tocommunicate using a Wi-Fi protocol (e.g., 802.11(n)) or another protocolsuch as Zigbee, Z-Wave, MICS, or Bluetooth. In various examples, thesystem 1200 may use the communication circuit 1212 to communicate withone or more system components (e.g., integrated displays, lightfeatures, motors, etc.) or with a second manipulating system or with auser control unit via wired or wireless communication connection.

The system 1200 may include a memory circuit 1214 and a non-volatilememory 1216, both of which may also be centralized or distributed.Various information such as application or operating program, userpreferences, user profiles, heights (e.g., user height), calibrationinformation (e.g., sensor calibration data), system tuning or operationinformation, poses (e.g., pose configuration information such as IV,transport, ready to drape, and ready to dock poses), historical useinformation, session information (e.g., duration of a surgical session,or movement or orientation or position history), login or authentication(e.g., username and password) information, or other information may bestored in the memory circuit 1214, non-volatile memory 1216, or both.

The system 1200 may include one or more motion sensors 1218 or forcesensors 1220, or both, or similar user presence or input sensors, any ofwhich may, for example, provide input for control of light features,motors (described below), or other system components or features.

The system 1200 may include a link motor 1222, which may control amovement of a link or other component, such as a translational movementof a link with respect to a component (e.g., column), or a translation(e.g., telescope) of a column with respect to a base.

The system 1200 may include one or more joint motors 1224 (e.g., onejoint motor for each degree of freedom of each joint of an arm), whichmay, for example, be configured to adjust an orientation of a jointresponsive to a command received from a processor (e.g., processor1250). For example, a joint motor 1224 may change a relative orientationof two links, e.g., rotate an arm link with respect to an adjacentcomponent. As an example, a joint motor 1224 may rotate a forearm asdescribed above.

The system 1200 may also include one or more wrist motors 1226. Forexample, a system 1200 may include a first wrist motor configured toadjust a pitch of a spar relative to a forearm link and a second wristmotor configured to adjust a yaw angle of an instrument mount relativeto the forearm link.

The system 1200 may include one or more base motors 1230 that may beconfigured to move a base. For example, a base motor may be configuredto roll, turn, or otherwise move a wheel, roller ball, or othercomponent to drive the base across the floor or otherwise move the baseas described herein. In some examples, a system may include one basemotor coupled to each wheel or roller ball, or a motor may be coupledthrough a drive system (not shown) to two or more wheels or rollerballs.

The system may include a handle motor 1232, which may be configured tochange an angle or orientation of a handle or extend or retract a handleas described herein.

The names of the motors are arbitrary and are provided for purpose ofexplanation. For example, a system may include a motor configured toadjust a pitch angle or yaw angle of an instrument mount that is not inor at a wrist.

Any or all of the motors or light features or displays may becommunicatively coupled to a processor (e.g., processor 1250) and may beconfigured to execute instructions or respond to signals from theprocessor (e.g., move or lock or change speed or light up or dim orchange color or pattern).

The system 1200 may also include one or more power sources such as abattery or capacitor or AC power supply (e.g., configured to connect toa wall outlet). A battery 1234 may provide power to the light features,motors, sensors, processors, or other components. The battery 1234 maybe rechargeable. For example, an AC power supply 1238 may provide powerto a rechargeable battery 1234 that may provide DC power to any of thecomponents described above. Alternatively, the system 1200 may run on ACpower. In an example, the AC power supply may step down wall outletpower to provide low voltage AC power to lights, motors, or othercomponents. In some examples, the system 1200 may include one or moreisolation circuits 1236 that may isolate the system 1200 from a linepower source or wired data connection, which may protect the patientfrom a power surge through a power line or data line (e.g., networkcable).

Light Features

One or more light features such as a lighted ring may be incorporatedinto an arm. For example, a light feature may be incorporated as a thinring at a joint between two links (e.g., joints as shown in FIGS. 2A-2I,3C, 5A-5B, 6A, 7A-7B, 8, and 9A-9D). A light feature may also be a widerring. A light feature may include a light pattern, light colorvariation, or animation. In some examples, a light feature may be ashort link, or it may be part of a short link, which may be between twomain structural links of an arm or other manipulating system assembly. Alight feature may be, or be part of, a handle, helm, or othermanipulating system component. A lighted ring may be a dedicated displaylink as described above. Alternatively, a lighted ring may be output ona controlled display, such as a controlled display on a dedicateddisplay link (see e.g., FIGS. 4A and 4B, display 402).

A light feature may be controlled locally by a processor in amanipulating system, or it may be controlled by a separate computerprocessing system that also controls other devices or features in ateleoperated surgical system (e.g., user control system 150 or auxiliarysystem 175 shown in FIG. 1 ), or by a combination of a computer in themanipulating system and a second computer. A light feature in a modularmanipulating system may be controlled in coordination with a largersystem (e.g., with da Vinci Xi® surgical system).

In various examples, a light feature may be controlled responsive to auser input (e.g., from a clinician), or by a processor responsive to amanipulating system event, state, or operating mode. A light feature mayprovide a communication to a clinician or amongst clinicians (e.g., acommunication from a surgeon to a nurse, or vice versa, during ateleoperated surgical procedure). A light feature (e.g., rings) mayinclude various visual aspects, which may communicate a status,identity, alert, instruction, or other information to the user oranother person associated with operating a manipulating system. In someexamples, a system may include a light sensor, and the brightness, coloror other appearance of a light feature may be adjusted based at least inpart on input from the light sensor (e.g., adjusted based on a level orquality of ambient lighting).

Example Light Feature Patterns

Light features and patterns displayed by light features are generallyarranged in accordance with the shape of the object on or about whichthey are located. As examples, various light features and patternsdescribed herein are in accordance with circular objects in support ofthe various described human-engineered manipulating system features.Other light feature shapes and patterns may be used, however, to closelyvisually integrate a light feature with the overall human-engineeredconsiderations of a manipulating system component or location. Forexample, light features may be generally rounded (e.g., circular, oval,etc.) or generally polygonal (e.g., square, hexagonal, octagonal, etc.)rings. Light features may be closed or open geometries, and they mayextend more than 360 degrees around an object. In some examples a lightfeature may extend around two or more objects, such as around twocolumns extending from a base. And, light features may be linear orcurvilinear, such as in association with translational motion of anobject. Further, two or more light features may be combined to form asingle light features (i.e., a light feature may include two or morediscrete subfeature components). Visual patterns displayed by lightfeatures are in accordance with the shape of the light features (e.g., acurving pattern on a rounded light feature in association with arotational motion, a linear pattern on a straight light feature inassociation with translational motion, etc.). The following descriptionof light feature visual patterns applies to both a single light featureand to a light feature with discrete subfeature components, both ofwhich may display a visual pattern that may be continuous (e.g., acontinuous line) or that may include discrete visual elements (e.g., adashed or dotted line). Therefore, various visual patterns aredescribed, and skilled persons will understand that the visual patternsdescribe light features of both types, so that the term light featurepattern applies to both the visual display perceived by a viewer and thephysical arrangement of a particular light feature.

FIG. 13 is an illustration of aspects of example light feature patterns,which may be applied to light features such as rings. As shown in FIGS.2A-2J, 9A-9C, and other FIGS., a light feature may include a light ringpattern, which may extend circumferentially part way or all the wayaround an interface portion of a teleoperated surgical system.

Various light feature shapes and visual patterns are possible, and theaspects of the light feature pattern may communicate information to auser. Light features and visual patterns may extend directly across acomponent, or they may extend at an oblique angle (such as an obliqueangle relative to a long axis of an arm link). FIG. 13A shows an examplelight feature pattern 1302 that extends at an oblique angle around asystem component (e.g., arm) 1303. FIG. 13B shows a light featurepattern 1304 that includes discrete subfeature patterns that extendaround a system component 1305 in an oblique angle pattern similar tolight feature pattern 1302. As shown, the light feature pattern 1304includes subfeature patterns that are short, straight lines (“dashes”)aligned with component 1305's long axis. A light feature may extendhelically around a system component. FIG. 13C shows an example lightfeature pattern 1306 that includes dashes similar to those described forlight feature pattern 1304 and that extend around a system component1307 in a helical pattern larger than 360 degrees.

In some examples, a light feature pattern may transition from a solidfeature pattern (e.g., continuous ring) as shown in FIG. 13A to aninterrupted feature (e.g., dashed ring) as shown in FIG. 13B to indicatea state change, such as a clutch mode (described below) or movementstate (e.g., locked or unlocked joint). In some examples, a lightfeature pattern may transition from a single line or set of dashed linesto a double set or continuous helix as shown in FIG. 13C to communicatea state change (e.g. change in range of motion or motion speed ordegrees of freedom).

In some examples, a size of a light feature pattern (or size ofilluminated portion of a light feature pattern) may change tocommunicate information to a user. For example, FIG. 13D shows acomponent 1309 with a light feature pattern 1308 in a first state inwhich the light feature pattern has a first (small) dimension, FIG. 13Eshows the light feature pattern 1308 in a second state in which thelight feature has a second (intermediate) dimension that is larger thanthe first dimension, and FIG. 13F shows the light feature pattern 1308in a third state in which the light feature has a third (large)dimension that is larger than the first and second dimension. The first,second, and third states may communicate, for example, an amount ofdisplacement (e.g., translational displacement or angular displacement)of a joint. The light feature pattern 1308 may also be animated orchange color, e.g., the length of the segments in the light featurepattern 1308 may grow longer (e.g. progress smoothly from the stateshown in FIG. 13D to 13E to 13F, or vice-versa) as a joint moves. Insome examples, the length of the segments may indicate translation of acomponent (e.g. telescopic movement of one part relative to another),and a rotational movement or color or other visual state change mayindicate rotation.

In some examples, different aspects or elements of a light feature mayindicate different quantities or types of information about movement ofa joint near the light feature, such as rotational and translationmotion at the joint, or range of motion. In light feature pattern 1308,for example, successive dashed subfeature patterns may illuminate toindicate a rotational joint displacement, and the subfeature patternsmay also increase in length to indicate a translational jointdisplacement. And, light feature pattern 1308 as a whole, or particularsubfeature patterns, may change color, blink, and the like to indicateinformation such as joint ROM limit, target joint displacement formanual movement, manipulator system or component status, etc. Persons ofskill in the art will understand that various light feature patterns maybe similarly modified, such as a light feature ring pattern illuminatingalong its arc to indicate rotational joint displacement and becomingwider to indicate translational displacement, optionally changing coloror color pattern in association with rotational or translationaldisplacement to convey information to the user. And as shown in FIGS.13D-13F, a linear light feature pattern change may be associated with arotational joint displacement (e.g., longer corresponds to position inrotational ROM).

FIG. 13G shows component 1311 with an example light feature pattern 1310that has a first subfeature pattern element 1312 (e.g., bars) and asecond subfeature pattern element 1314 (e.g., dots). In an example, thefirst element 1312 (e.g., bars that increase and decrease in length) maycorrespond to a first type of motion (e.g., translation in the directionof the bars) and the second element 1314 may correspond to a second typeof motion (e.g., rotation around the direction of the dots). An aspectof the first element 1312 (e.g., length of bars or thickness of bar orcolor or brightness or position) may change to indicate an amount ofmovement or a rate of change of position (e.g., speed of movement). Anaspect of the second element 1314 (e.g., size of dots or color orintensity or position) may change to indicate an amount of movement orrate of change of movement.

FIG. 13H shows a component 1313 with an example light feature pattern1322 having a displacement light feature pattern 1316 (e.g., one or moredisplacement bars) that may indicate an amount or rate of movement(e.g., length of displacement bar(s) may indicate an amount ofmovement). A first range of motion light feature pattern 1318 mayindicate a first range of motion limit for a joint. A second range ofmotion light feature pattern 1320 may indicate a second range of motionlimit for the joint. The first and second range of motion limits may beopposite ends of the same range of motion for the joint (e.g., top andbottom of a translational displacement), or they may two different typesof motion (e.g., rotational and translational) for a multi-DOF joint. Inan example, the length of the displacement feature 1316 may increase(e.g., displacement bars get longer) as a component or portion of asystem (e.g., arm link or arm joint) moves, and first and second rangesof motion (e.g., rotational limits or translation limits) may beindicated by relative position of the ends of the feature (e.g., top orbottom of the bar) relative to the range of motion features 1318, 1320.In some examples, an aspect of one or more of the light features 1316,1318, 1320 may change color, brightness, or intensity, or start to flashto indicate that a movement is near an end of a range of motion (e.g.,yellow) or at an end of a range of motion (e.g., red).

Any of the examples shown in FIGS. 13A-H may be combined together (e.g.,angled examples may have variable width or length), or they may becombined with a controllable color state (such as the examples describedbelow) or animative state (e.g., light feature pattern motion mayindicate joint motion or flashing may indicate a state or state change).

The aspects of light feature patterns shown in FIGS. 13A-13H anddescribed below may be combined with any of the systems, methods, andfeatures described above or below. The example descriptions below aredescribed as “rings” merely as illustrative examples. The aspects ofring examples apply to other light feature pattern shapes. One aspect isthe way the light features are integrated with the design of the arm,e.g., mechanically seamless, and visually positioned at a location onthe arm relevant to the information the light feature is communicating.For example, a light feature (e.g., ring) may be positioned at aninterface between components (e.g., at a joint between a link and anelbow or another link.

Light Features on an Orienting Platform

FIG. 16 is an illustration of an example manipulating system 100modified with one or more light features. As shown, four manipulatorarms are coupled to a single orienting platform, which is in turnsupported by another arm that extends from a mechanically grounded base(not shown). The manipulating system 100 may be the system 100 shown inFIGS. 1A and 1B, modified as described herein. The manipulating system100 may include a light feature 1602 at a flex joint 1612 where a firstarm 1622 meets an orienting platform 1620. The flex joint 1612 may be ajoint where an arm couples to the orienting platform 1620, and it may bea pivoting joint that allows the arm to rotate around the orientingplatform 1620. Other types of flex joints are also possible(translation, spherical, etc.).

The light feature 1602 may align with (e.g., extend around) the flexjoint 1612 on an underside 1610 of the orienting platform 1620 so thatit is visible to a clinical user looking up at the orienting platformwhile moving the arm 1622. The light feature 1602 may, for example,extend partially or fully around a pivot part that couples the arm 1622to the orienting platform 1620.

In some examples, an aspect of the light feature 1602 may indicate acharacteristic of the arm position, such as a desired joint displacementlocation, an undesired joint displacement location, or a moderatelydesired or acceptable joint displacement location. An a light featurepattern appearance scheme may include a number of appearance states thatcorrespond to the desirability of arm placement. For example, the system100 may use a color-coded scheme (e.g., with color-changing LED panel),where red indicates an undesired placement, yellow indicates amoderately desired placement, and green indicates a desired placement.

In some examples, the light feature pattern 1602 may have a consistentvisual appearance throughout the light feature 1602. For example, thelight feature 1602 may be green, or may be yellow, or may be red. Thevisual appearance may be based upon the position of the correspondingarm. For example, if the arm 1622 is in a desirable position, the lightfeature 1602 may have a first appearance, and if the arm 1622 is in anundesirable position, the light feature 1602 may appear red. The lightfeature pattern may change based upon movement of the arm. For example,responsive movement of the arm 1622 from an undesired position to amoderately desired position, the light feature pattern 1602 may changefrom the first appearance (e.g., red) in the undesired position to asecond appearance (e.g., yellow) in the moderately desired position. Andresponsive to further movement of the arm 1622 to a desired position,the light feature pattern 1602 may change from the second appearance(e.g., yellow) to a third appearance (e.g., green) in the desiredposition. The correspondence of visual appearance to arm position mayguide a user in choosing an arm position based at least in part ondesirability of the position indicated by the light feature.

In some examples, the light feature pattern 1602 may have an appearancethat is not the same throughout the light feature. For example, avariation appearance (e.g., color difference or brightness) across alight feature may communicate a desirability of a range of positions.For example, a first portion 1636 of the light feature 1602 may have afirst appearance (e.g., red), which may indicate that the location ofthe first portion 1636 corresponds to an undesirable position (e.g.,potential collision with another arm coupled to the orienting platform,or another arm from a second manipulating system), a second portion 1638of the light feature 1602 may have a second appearance (e.g., yellow),which may indicate a moderately desired (e.g., acceptable but not ideal)position, and a third portion 1640 of the light feature 1602 may have athird appearance (e.g., green) which may indicate a desired position. Insome examples, the light feature pattern may include additional discretelight portions (e.g., six, twelve, eighteen, or more), which may eachhave a different appearance (e.g., e.g., shades of green, yellow, orred, or blends thereof).

The first arm 1622 may include an indicator 1642, which may be near thelight feature 1602 to allow for observation of the relative position ofthe indicator relative to the light feature 1602. The indicator 1642 maycorrespond to a portion on the light feature pattern, which may indicatethe desirability of the position of the arm 1622. For example, theindicator 1642 may align with a portion of the light feature 1602 havinga second appearance (indicating a moderately desired position), andmovement of the arm 1622 counter-clockwise (as indicated by the arrow)may move the indicator 1642 toward the third portion 1640 of the lightfeature 1602. The alignment of the indicator 1642 with the third portion1640 may indicate that the arm 1622 is in a desired position (e.g., aposition with a low risk of collision).

In various examples, the indicator 1642 itself may be light feature, orit may be unlighted. In some examples, the indicator 1642 may have avariable light feature pattern appearance that may be controlled matchthe appearance of the indicator 1642 to the appearance of acorresponding portion on the light feature pattern 1602 in order toreinforce the information conveyed to the user. For example, when theindicator 1642 is in a “green zone” (adjacent a green portion of thelight feature 1602), the indicator 1642 may be green, and when theindicator 1642 is in a “red zone” (adjacent a red portion of the lightfeature 1602), the indicator 1642 may turn red.

The system 100 may include a second light feature 1604 at a second flexjoint 1614 for a second arm 1624. In some examples, the desirability ofan arm position and the corresponding appearance a light feature (e.g.,green, yellow, red) may depend on a position relative to an adjacent arm(e.g., the appearance of the second light feature may depend upon theposition of the second arm 1624 relative to the first arm 1622).

In some examples, the indication of desirability of an arm position maybe based at least in part upon the potential for a collision with anadjacent arm. In this context, a collision may not be a physicalcollision (which may be prevented by the system or a related controlsystem), but rather may be additionally or alternatively include a statein which one or both arms are prevented from movement in a direction toavoid a possibility of a physical collision. A desirable position of anarm may correspond to a position in which the arm is unlikely or leastlikely to experience a collision with an adjacent arm (e.g., based uponsatisfaction of one or more collision avoidance criteria, when acalculated collision potential is below a threshold), and an undesirableposition may correspond to a position in which a collision is somewhatmore likely than in the desirable position (e.g., one or more collisionavoidance criteria not satisfied, or a calculated collision potential isabove a threshold). The potential for a collision may be based on one ormore collision avoidance criterion, such as a spacing of adjacent arms(and optionally including spacing of attached instruments) or ananticipated range of motion needed to perform a procedure.

Thus the information conveyed by various light feature patterns on amanipulating system may be dynamic, so that a first light featureconveys information not solely about an associated system component(e.g., a first joint or other corresponding system component), but alsodepending on the state of one or more second components in themanipulating system or in a second manipulating system. And, a lightfeature corresponding to the one or more second components may alsoindicate that those second one or more other components are associatedwith the information conveyed by the first light feature. For example,if a first arm is in position to collide with a second arm, a firstlight feature associated with the first arm indicates the potential armcollision problem by flashing, then a second light feature associatedwith the second arm also flashes to alert the user that the second armis the potential collision problem. Collision potential is merely anexample of the many dependent states between two or more systemcomponents that may be indicated by light features associated with eachof the two or more system components. This visual information assiststhe user to determine which system components are involved with aparticular situation so that the user can modify a system configurationaccordingly.

The system may 100 also include a third light feature 1606 at a thirdflex joint 1616 for a third arm 1626, and a fourth light feature 1608 ata fourth flex joint 1618 for a fourth arm 1628. The third light feature1606 and fourth light feature 1608 may indicate a status or state (e.g.,desirability of position) as described above.

In some examples, a light feature may indicate desirable of positionrelative to two adjacent arms. For example, a first portion 1630 of thesecond light feature 1604 may indicate the desirability of the positionof the second arm 1624 relative to the first arm 1622, and a secondportion 1632 of the second light feature 1604 may indicate thedesirability of the second arm 1624 relative to the third arm 1626. Inan example situation, the first portion 1630 may be red, indicating thatthe first arm is too close to the second arm, and the second portion1632 may be green, indicating that the third arm 1626 and second arm1624 are in a desirable placement (e.g., unlikely to experience acollision). In such a scheme, a user may be guided by the light featuresto position arms so that each light feature indicates a desirableposition (green) or acceptable position (e.g., yellow) throughout thelight feature.

While various examples above have been described in terms of a colorscheme (e.g., red, yellow, green) for purpose of explanation, otherlight feature pattern schemes are also possible, such as a brightnessscheme (e.g., bright indicates desired placement and dim indicates anundesired placement) or a flashing or pulsing scheme (e.g., fastflashing indicates desired placement and slow flashing indicatesundesired placement) or a motion scheme (e.g., light feature patternanimation movement in a clockwise direction indicates the desiredposition is in a clockwise direction, in a counter-clockwise directionindicates the desired position is in a counter-clockwise direction, nomovement indicates the desired position, and the speed of animatedmovement may optionally indicate a proximity to the desired position).In some examples, two or more light feature pattern schemes may becombined (e.g., a color scheme may be combined with a motion scheme or aflashing or pulsing scheme). In some examples, the scheme may beuser-configurable. While the light features have sometimes beendescribed as having discrete portions for the purpose of explanation ofthe examples illustrated by FIG. 16 , any of the light featuresdescribed herein may also be continuous, or may appear to be continuous(e.g., using a multiplicity of discrete LEDs or other light elements),and the light feature may have discrete regions with differentappearances (as described above), or the light feature may appear tohave a continuously changing appearance across its length (e.g.,gradually blending from green to yellow to red).

Physical Construction and Appearance of a Light Feature

A light feature may include a plurality of lights that together form alight feature, such as a light ring. For example, a light ring mayinclude a number (e.g., forty (40)) of light emitting diodes (LEDs),which may be under a cover (e.g., transparent, translucent, ormicro-drilled cover), and may form a light feature having either acontinuous or a segmented appearance. In some examples, a light featuremay include LEDs of different colors to enable changing of the color ofthe light feature, or to enable a variation in the color across a lightfeature (e.g., one portion of the light feature may be red, a secondportion of the light feature may be green, and a third portion betweenthe first and second portions may be yellow).

A light feature may be designed to “disappear,” as described above withthe integral display. For example, a ring may be formed from atranslucent material (e.g., plastic or CORIAN®) and has an appearancethat matches the adjacent structure when not lit. In another example, aring may be formed from a plurality of small holes that transmit light.The holes may be in front of a light display (e.g., LED array). Aprocess such as vacuum metalizing may be used to give ring a metallicappearance when not lighted, which may also be used to integrate thevisual appearance of the unlit ring into the human-engineered overallvisual appearance of the manipulating system.

A light feature may be formed in the shape of a ring that extends atleast part way around a portion of a medical device. A light ring is notnecessarily circumferential. For example, a light ring may becircumferential, helical, a double-helix, or asymmetric. In someexamples, a light ring may be or include one or more arc segments. Invarious examples, a ring may be thicker (e.g., 2 cm or 5 cm or 10 cm) orthinner (e.g., 1-3 mm) than shown in FIGS. 2A-J or 9A-9C.

A light ring (or other light feature) may extend fully around aninterface or arm, or a ring may extend only partially around an arm orinterface. In various examples, a light ring may be continuous orbroken. For example, a ring may appear to be a single continuous strip,or a ring may be or appear to be a series of discrete lights. Forexample, a ring may be formed of a plurality of dots, dashes, lines,icons, or other patterns. In some examples, discrete light elements thatform a ring may sequentially light up so that one or more lightedportions appear to move along the ring (e.g., run around an interface ortransverse to an interface).

In some examples a light ring may extend around a portion of amanipulating system base, or all the way around (as shown, for example,in FIG. 9A), so that the light ring is at the interface between the baseand the floor or other mechanical ground. Such a base-ground interfacemay be used to indicate information such as “system is ready to move” or“system is braked to remain in place”. And, lighted portions of abase-ground interface may indicate directional movement such as flashinggreen in a particular direction and solid red in other directions.Similarly, a light ring may be at the bottom of column extending from abase (as shown in FIG. 9A), or at the top of a column where amanipulating arm is coupled (as shown in FIG. 2A).

In some examples, a light feature may be a touch point or control inputdevice (e.g., touch sensitive button), or it may be adjacent to orintegrated with a touch point or control input device.

In some examples, an input device such as a control button may beintegrated into a light ring or other light feature. For example, alight ring may include a real (e.g., button) or virtual (e.g., touchscreen) input that is configured to receive an input from a user, suchas a button press, finger slide, or other touch input. In variousexamples, a manipulating system operating mode, arm clutch state, ordirection of movement may be input by using the input device.

Light features may be set off from other structures by a visual boundarymember, such as a metal ring (e.g., a visual appearance consistent witha the visual appearance of other significant locations (e.g., handle) onthe manipulator system so that the human-engineered visual appearance of“this is an important location” is consistently communicated to a user).For example, FIGS. 20A-20B show an illustration of a medical device2000, such as a manipulator arm, in which a second arm portion 2014 maybe visually set off from a light feature 2012 by a first border 2004.The first border 2004 may, for example, be a metal ring between thelight feature 2012 and the second arm portion 2014. The device 2000 mayinclude a second light feature (e.g., ring) 2010, which may be visuallyset off from the first light feature 2012 by a second border 2008. Thefirst border 2004 and second border 2008 may be, for example, be formedof machined polished aluminum and may optionally have an appearanceconsistent with the polished handles described above.

In some examples, a controllable display may output a light featurepattern similar to the physical characteristic of another light feature.As shown in FIG. 4B, for example, a light strip may be illuminated onthe display to simulate the appearance of a light feature ring, whichoptionally is at another location on the system. In this way two or moredifferent light feature types may be configured to output a consistentlight feature pattern in a system, and so ensure consistenthuman-engineered user interaction features in the system.

Visual Aspects of Light Features

Various visual light pattern features may be applied to a light feature.For example, a light feature pattern may change color, brightness, orshape. In some examples, a light feature (e.g., ring) may include atime-varying feature. For example, a light feature may change color orbrightness, or a light feature may flash, pulse, “breathe” (e.g., slowlyshift brightness on and off), or display according to a time pattern. Insome examples, a light feature pattern may pulse with discrete on/offpulses, which may follow a continuous or intermittent square wavepattern, and which may create a strobe or flashing effect. In someexamples, a light feature pattern may be controlled to provide acontinuous transition from bright to dim or off (e.g., follow a sinewave or saw tooth pattern), which may create a “breathing” visualeffect.

In some examples, a light feature pattern may flash, pulse, or otherwisechange to indicate direction (e.g., to indicate a rotation ortranslation). For example, a light feature pattern may indicate adirection by sequentially lighting one or more segments (e.g., LEDs orgroups of LEDs) to create the appearance that a lit portion moves alongthe light feature, or that one or more lit portions follow (“chase”) alead lit portion that moves along the light feature.

When a system is powered on, a startup effect may be presented on all orsome of the light features of the system. For example, two or more lightfeatures may be lit in sequence (with overlapping or non-overlapping littimes) along the system for a defined duration, which may create anappearance of energy or light traveling or “washing” over the system,and may appear to “bring the system to life” as the system starts up. Invarious examples a color change or pattern, brightness change orpattern, pulse pattern (e.g., one or more flashes), or any combinationthereof may be used to indicate system startup. The completion of suchan effect may indicate that a startup process is completed (or hasstarted).

In some examples, a device may include one or more sensors configured tosense a user's presence (e.g., a contact, proximity, camera, motionsensor, or other presence sensor), and responsive to detection of a user(e.g., the user's hand on or near the light feature), the appearance ofa light feature may change (e.g., brighten or change color). In variousexamples, the sensor may be next to, on, or under a light feature (e.g.,so the system can detect a user's presence near the light feature). Insome examples, a light feature may turn on, brighten, or change colorwhen a user is detected. In some examples, a light feature near a usercontrol input may light up to alert the user of proximity or contactwith the control input. For example, as shown in FIG. 17 a light feature1702 may turn on as a user's hand comes close to control input 1704, andlikewise light feature 1706 may turn on as the user's other hand comesclose to control input 1708. In this way the user is alerted that thesystem is operating and is ready. Optionally, a light feature may turnon in one color (e.g., red) if the system is not ready to receive aninput at the associated control input device and in another color (e.g.,green) if the system is ready to receive an input a the associatedcontrol input device. In some examples, a light feature may changeappearance to alert a user that he or she is near a pinch point or otherpotentially injurious location. For example, a system may use jointsensors, awareness of the position or orientation of system components(e.g., arm links) to determine the location of a pinch point (e.g., apinch within an arm, or a pinch between an arm and another object suchas a surgical table), and the existence or location of the potentialpinch point may be identified with a light feature (e.g., identified bycolor or other appearance change continuously, or responsive todetection of a user near the potential pinch point).

In some examples, a single light feature may include one or moresections that are visually different (e.g., brighter or a differentcolor) from other portions of the light feature, as shown for example inFIGS. 20A and 20B. In some examples, the visually different section maychange color, or move in the direction of a detected touch point, orboth.

In some examples, a system may detect when a user is near the system ora specific part of the system. Responsive to detection of a nearby user,a system may increase the brightness of a light feature. For example,when a user is close to a system, the system may increase the brightnessof one or more lights that are closest to the user to illuminate a workarea on the system or nearby.

A system may control light features in response to detected sound. Forexample, a system may change the appearance (e.g., brighten or changecolor) of a light feature in the proximity or direction from which itsenses a sound, such as a voice command. Indicating a direction of areceived voice command may inform a user which of a plurality of soundsor voices the system is treating as a command or input. For example, ifa first user is at a first side of a system and a second user is at theopposite second side of the system, and both users are giving voicecommands to the system, one or more light features light or otherwiseindicate which user's voice is being received as a command input.

Light feature patterns may be customized or personalized. For example, auser's color preference, brightness preference, or motion preference maybe applied and saved for one or more light features. In some examples, aspecified light feature may be defined for an identification purpose.The identification light ring optionally may not be used for otherusability features (e.g., a particular ring may optionally not be usedto communicate system status). An identification light feature mayfacilitate identification of an individual system in a facility such asa hospital (e.g., a system that has a green identification light featuremay be distinguishable from a system that has a blue identificationlight feature). In some examples, an identification light feature mayassociate a system with related systems or sub-systems. For example, acontrol unit, manipulator unit, or auxiliary unit may be configured towork together as a system and are therefore visually identified asrelated based on an identification light feature (e.g., two or morerelated units may present a specific color, such as green, using theidentification light feature). In some examples, two units may bemodular or usable with various systems, and may be paired or otherwiseconfigured to work together, and the two units may present an identicalor similar appearance (e.g., blue) on a light feature to indicate theirrelated status. In some examples, a manipulating system identificationlight feature may change color or otherwise visually indicate a changeof control from a first control unit to a second control unit. Forexample, an identification light feature may indicate which of twosurgeons is controlling a particular manipulator arm.

Communication Using a Light Feature

In some examples, a visual aspect of a light feature such as ananimation, pattern, brightness, or color may indicate information abouta system, such as a status (e.g., joint or arm range of motion, systemis ready for operation), or movement (e.g., to indicate that a portionof a system is moving or about to move or a direction of movement), oralert a user to a problem (e.g., a potential collision, a range ofmotion problem, or a fault condition).

In some examples, one or more light features on a column or other systemlink may simultaneously indicate both rotational orientation andvertical translation. For example, a display characteristic along thelength of the light feature may display rotational information, and adisplay characteristic along the width of the ring displayingtranslational information. (See e.g., FIGS. 13B-13H, described above.)

In some examples, a ring (or other light feature) may be more than onecolor, and information may be indicated by color. For example, one ormore rings (or other light features) may be different colors ondifferent sides of an object (e.g., on a base, on an interface portionof an arm) or facing different directions. For example, the system shownin FIG. 9A may have a first color (e.g., green) at a portion of a lightring portion at a front of base, and blue light ring portion at side ofbase, and a red light portion at a rear portion of a base. The colorsmay indicate a front portion (e.g., indicated by green) and a rearportion (e.g., indicated by red) of the base. In other examples, a colormay indicate a direction of permitted movement, e.g., a first color(e.g., green) may indicate a permitted direction and a second color(e.g., red) may indicate a direction that is not permitted. In someexamples, a first color may indicate a first direction is good (e.g.,indicated by green) and a second direction is not good (e.g., indicatedby red) or not desired or not pending or not needed.

A light feature may provide an instruction on direction of movement. Forexample, a color or pattern or animation may indicate to a user that abase or arm link or joint or other portion of a system should be movedin a direction indicated by the light pattern (e.g., move in directionof green light or not in direction of red light).

A light feature may indicate (e.g., provide an alert) pending orupcoming movement or a nature of a movement. For example, a color orflashing light may indicate that a movement of a joint is pending or mayindicate a direction of pending movement. A light feature or combinationof light features may indicate a combination of capability, nature, andpendency of movement of a joint, link, or arm component. In variousexamples, a light feature may indicate a joint is able to move, or ajoint is actively moving, or that movement of a joint is pending, ortranslational motion or rotational motion of a link is pending, orcombined rotational and translational motion is pending.

In some examples, a light feature may indicate a joint position orproximity to one or more joint range of motion limits. In some examples,a moving strip or other indication within a lighted ring may show jointposition within a range of motion. For example, a light feature maychange color near joint a range of motion limit (e.g., green or whitemay indicates not near a limit; yellow may indicate that a system isclose to a range of motion limit; and red may indicate that a system isat a range of motion limit).

In some examples, a light feature may indicate a joint position relativeto a target displacement or target position. For example, a lightfeature may display an indication of a target and a current displacementlocation (e.g., angular displacement location), and a user may beprompted to move the joint to align the current displacement locationindicator with the target displacement location indicator (e.g., theuser's goal may be to move the current displacement indicator to matchthe target indicator). In some examples, a light feature may provide anindication of how to move a joint to reach a target joint displacement.For example, a light feature may display an arrow indicating a directionof movement. As another example, a light feature may display a barbetween a current position and a target position or displacement, and auser may be prompted to make the bar smaller by moving the joint towardthe target orientation. In some examples, animated light patterns maymove to indicate a direction of movement. A light feature may indicateproximity to a target. For example, the light feature may use a colorscheme such as orange to mean farther from a target displacement, yellowto mean closer to the target, yellow-green to mean “almost at thetarget.” and green to mean that a joint is at the target displacement.Other color schemes and light feature patterns are also possible, suchas brightness (e.g., brighter means closer, dimmer means less close totarget) or a flashing behavior (e.g., faster flash means closer, slowerflash means further, steady light means the joint is at the target).

In some examples, a light feature may indicate a safe grab point on adevice or system (e.g., green may indicate a safe grab point), or mayindicate not to touch an arm (e.g., red indicates not to touch an arm ora portion of an arm).

In some examples, a system may use color coding to assist withidentification or differentiation of units or links. For example, onemodular system may have lights of a first color (e.g., blue lights), anda second system may have lights of a second color (e.g., yellow lights).Or, for a single modular system, an arm or other portion may have lightsof a first color (e.g., blue), and a second arm or other portion mayhave lights of a second color (e.g., yellow).

In some examples, a light feature (e.g., steady red or flashing red) mayindicate a location of an emergency handle or an instruction to pull anemergency handle or to initiate another emergency operation, or it mayindicate the system performing an emergency action.

In various examples, a light feature may indicate a non-recoverablefault or a system power up or power down (e.g., an animation such as anapparently slowly moving light to indicate power up, or a slowly dimminglight to indicate power down).

In some examples, a light feature may indicate a system operating mode,e.g., a following mode (“following” means an instrument motion follows auser control system input device movement), a control mode, a clutchmode (e.g., in which a joint brake is released to allow manual movementof a joint), a break-away clutch mode (e.g., in which a joint willresist movement from a force/torque on an associated link until athreshold force/torque is reached, at which point joint brake willrelease, further described below), a standby mode, a transport mode, oranother system operating mode.

In various examples, a light feature may also be used to communicate acollision or near collision (e.g., a joint motion lock to avoid acollision) of a portion of a system (e.g., an arm) with another object(e.g., a surgical table or another arm). In some examples, a collisionor near collision may be determined based at least on sensorinformation, such as information from a force sensor, an object ormotion sensor (e.g., infrared sensor), or a proximity sensor (e.g.,capacitive sensor). In some examples, a visual aspect (e.g., color) of alight feature may communicate that a portion of a system (e.g., an arm)is nearing (e.g., yellow) or at (e.g., red) end of joint or arm linkrange of motion.

In some examples, an aspect of one or more light features may indicate astate or status of the system, such as a ready state (“I am ready”),which may for example be indicated by a dim light feature, such as a dimblue appearance, or may be indicated by a “chasing” effect in which alight feature appears to move in an orbital motion (e.g., around a lightring) or oscillating motion. A light feature may also indicate a controlstate (e.g., a state in which the system is being controlled), such asmanual control (which may for example be indicated by a steady brightappearance), or ready to be manually displaced (e.g., “clutched” brakerelease, which may be indicated for example by a pulsing or “breathingpattern”). A light feature may also indicate computerized control, suchas a teleoperated control in accordance with a user's input (e.g.,following movements of hand by a clinician), which may for example beindicated by a brightening of the light feature. A light feature mayalso indicate an automated joint displacement (e.g., carrying outautomated commands), which may for example be indicated by a pulsing or“breathing” pattern. A light feature may also indicate a systemself-test, during which a plurality (or all) of the light features in asystem may communicate (e.g., pulse) the self-test, or a light featureat a specific part of the system may communicate (e.g., pulse) when thatpart of the system is under self-test. A light feature may indicateengagement by a user. For example, a light feature may activate orchange (e.g., change color or brightness) when a user's hands aredetermined to be engaged with controls (e.g., based on detected contact,proximity, or presence). In some examples, a light feature may indicatethat a system is unlocked (e.g., a setup mode has been completed) andthe system is ready to operate. In some examples, a system may control alight feature to have a first response (e.g., no change, or change to afirst color) when a user input is engaged while a system is locked, andthe system may control the light feature to have a second response(e.g., change color or brightness) when user engagement is detectedwhile the system is unlocked.

A light feature may change appearance (e.g., change color, such as blueto yellow) to indicate the location (e.g., at a joint or part) of aproblem that needs to be addressed or corrected. In some examples, if anincorrect or unsafe action or state is detected, a light feature maychange appearance, such as changing appearance in a manner calculated toget a user's attention. For example, the light feature may flash quicklyor strobe or turn red.

A light feature may indicate a state of a particular part, joint, orconnection. For example, a light feature may indicate that an accessoryhas been correctly attached to a system. A correct connection may beindicated by a light feature near the connection or on the accessory, orit may be indicated by other light features on the system. A correctconnection may be indicated, for example, by a single pulse (shortseries of pulses), or a by a pulsing or “breathing” appearance, whichmay match a pulsing or breathing appearance of other aspects of thesystem, or may be present a different pulse appearance (e.g., faster orslower pulse rate).

In some examples, a light feature may indicate a joint state, such aswhether a joint is locked or fre to move (e.g., a clutch state of thejoint). For example, a clutch state may be indicated by light featurecolor (e.g., green indicates free, red indicates locked) or pattern(e.g., pulsing or flashing indicates a free state in which manualmovement is permitted, and steady indicates manual movement is notpermitted).

In some examples, a light feature at a connection interface (e.g., alight ring around the interface) may indicate a connection state. Forexample, a light feature may be in one state (e.g., dim or off) when noconnection is present and the light feature may be in a different state(e.g., bright) when a connection is present. In some examples, a lightfeature may briefly change appearance (e.g., pulse or flash) when aconnection is made. The connection interface may be a mechanicalconnection (e.g., an interface that couples two things together, such asa manipulator and a controller), or the connection interface may be anelectrical connection (e.g., a power connection or a communicationconnection). In some examples, a connection state, or change inconnection state may be accompanied by an audio signal to reinforce theindication regarding the connection state.

A light feature may indicate the state of a flux through a connection,such as ready to deliver the flux through the connection, proper flux ispassing through the connection, or insufficient flux is passing throughthe connection. For example, if a light feature is associated with anirrigation liquid or insufflation gas connection (e.g., a light ringaround the connection), the light feature may indicate green when theconnection is secure and the fluid is ready for delivery, may indicate apulsing green when a proper fluid flux is passing through theconnection, and may indicate a pulsing red when insufficient fluid fluxis passing through the connection. Other flux examples includeelectrosurgical energy (mono- or bipolar energy), vacuum suction,respiration gasses, and the like used during surgery.

Any of the light feature appearances or behaviors described herein maybe controlled by a medical device processor, which may receive one ormore inputs from a user or another device, and may send one or morecontrol signals to a light feature, or to a controller that may beconfigured to control a light feature, e.g., to control a voltage,current, or pattern delivered to a light feature. References to behaviorof a light feature may be interpreted as a system behavior based onprocessing in the medical device processor or controller. A lightfeature may change color, for example, based upon delivery of current toa colored light emitting diode. For example, a light feature may changefrom red to blue by reducing power to a red LED and increasing power toa blue diode, or a light feature may be changed to purple by activatingboth diodes (to combine the colors). A light feature may be made toappear to move by sequentially activating and deactivating adjacentLEDs, which may create a “chasing,” “running,” or oscillatingappearance.

Any of the light feature indications described herein may be accompaniedby an audio indication, which may reinforce the light feature indicationto the user, optionally by being synchronized to the corresponding lightfeature indication. The audio indication may include an aspect of asound, such as pitch (e.g., frequency), volume or dynamic (e.g.,amplitude), tone (e.g., fundamental or overtones), or a change orpattern in an aspect of sound. In various examples, an audio indicationmay include a rising, falling or pulsing volume (e.g., louder andsofter), or a rising, falling, or pulsing tone change, or a rising,falling, or pulsing pitch (e.g., changing frequency). In some examples,an audio indication may match a light feature indication. For example,the timing of a change in a light feature may match the timing of anaudio indication. Additionally or alternatively, a change in a qualityof an audio indication may correspond to a change in a light feature.For example, an audio indication may become louder or rise in pitch orchange tone as a light feature becomes brighter or changes color. Insome examples, the volume, pitch, or tone may pulse in synch with apulsing of a light feature. In some examples, a pitch of an audioindication may decrease as a joint (or object or part) moves toward atarget displacement.

In addition, the features described with respect to controllabledisplays (e.g., OLED displays) may also be applied to light features orcombined with light features (e.g., a light may indicate a status orstatus change and the status or status change may be indicated orexplained on an OLED display, or a light feature may indicate that anoption or feature is selectable on an OLED display.

Example Color/Pattern Schemes

A system may apply a scheme of colors and patterns to one or more lightfeatures to communicate the state of the system or to communicate a needfor action. A color scheme may, for example, be applied to one or morelight features in the system shown in FIG. 17-19 , or 1B and 16, or2A-2J, or other systems.

Colors and patterns may indicate system states. For example, when thesystem is in an off state, light features may be off and no pattern maybe present (because the light features are off), which may indicate thatthe system is not powered or is disabled. When the system is in a“ready” state, a light feature may be a first color (e.g., blue) and thelight feature may be dim, indicating that the system is ready for use.When the system is in a non-automatic movement state (e.g., clutched, asdescribed herein), the light feature may be the first color and bright,which may indicate that a user controlled, non-automatic action is inprogress (or available). For example, in a non-automatic movement state,a user may manually move a part of the system (e.g., an arm joint) withtheir hands, and the light feature may indicate the action in progress.When the system is in an automatic movement state, the light feature maybe the first color, and the light feature may follow a pattern (e.g.,the light feature may pulse), which may indicate that an automaticaction is in progress (e.g., an arm joint is moving in response to ateleoperated control signal, or a computer-determined action is inprocess). When the system is in a confirmation state, the light featuremay be the first color and may emit one or more pulses, which mayindicate a confirmation. For example, to confirm a state or receivedcommand, a light feature may flash once to provide a visual confirmationsignal to the user. When the system is in a “directional movement”state, the light feature may be the first color and follow a moving“chasing” pattern (e.g., portions of a light feature appear to move in adirection), which may indicate a directional movement, such as a pendingmotion or motion-in-progress, or a direction for a user to manually movea portion of the system (e.g., a joint). When the system is in a“potential action” state, the light feature may be a second color (e.g.,yellow) and emit a steady “solid” pattern (e.g., not changing, such asconsistently bright), or the light feature may follow a visual patternthat corresponds to another characteristic of the system (e.g., a readystate, or a non-automatic, automatic, or directional movement state).When the system is in an immediate action state, the light feature maybe a third color (e.g., red), and the system may emit a solid pattern(or pulsing pattern), which may indicate that immediate action isrequired. While blue, yellow, and red have been provided as examples,other combinations are also possible, and specific states or groups ofstates may have other color assignments (e.g., green may indicate readyand movement states may be indicated by blue, or automatic movement maybe indicated by purple). Other light feature pattern assignments mayalso be used.

Indication of Clutch States

In an example, the status of a break-away clutch feature may becommunicated by a light feature, an integrated display, or one or moreother features, alone or in combination. A system (e.g., system 200 orsystem 900 described above) may include a force sensor on a passivejoint. The system may include a sensor on an active joint that senses anattempt to move. When a sensed parameter (e.g., force) crosses athreshold, the system releases a clutch, and the joint is permitted tomove. When the joint stops moving, the system may re-apply the clutch(e.g., not permit the joint to move). The clutch release andre-engagement may be communicated through one or more light features.For example, from the user perspective, when the user pushes or pressesor pulls hard enough on a structural feature (e.g., on an arm or joint),the joint breaks away from its commanded teleoperated position andpermits manual movement, at which point a light feature may change(e.g., color or pattern or animation change) to indicate that statechange (e.g., break-away clutch point has been reached). The system maythen re-lock one or more joints when the user stops moving thestructural feature (e.g., after a predefined period of non-movement ormovement less than a threshold velocity), at which point the associatedlight feature pattern may change again. In another example, a system mayexit or pause a following mode when a joint clutch is actuated (activelyswitched or break-away) to enter a clutch mode that permits manual (orassisted) movement, and then the system may return to the following modeautomatically or in response to input after operating in clutch mode.These changes may be communicated via light features. Information onbreak-away clutching is found, e.g., in U.S. Pat. No. 9,452,020 B2(filed Aug. 15, 2013) and in U.S. Patent Application Pub. No. US2017/0172671 A1 (filed Mar. 17, 2015), both of which are incorporatedherein by reference.

Light Feature Indications in Guided Processes

Light features may be used to facilitate a guided process for adjustinga manipulating system, such as the system 100 or 200 or 900. In anexample, a light feature (e.g., light ring) at one or more specifiedjoints (e.g., key joints) may light up when the joint is properlypositioned or needs to be adjusted. For example, if a setup processneeds an arm to be moved from a present angular orientation (e.g., a90-degree orientation in which one arm link is perpendicular to a secondarm link) to a second angular orientation (e.g., a 45-degreeorientation), the light feature at the associated joint may becontrolled to present a specified appearance (e.g., bright, or changecolor, or pulse) to identify the joint to be moved. The light featuremay additionally or alternatively be controlled to indicate when atarget joint displacement has been achieved (e.g., by presenting aspecified color, such as green or blue). In some examples, lightfeatures at specified joints (e.g., key joints) may change color whenthe joints are positioned correctly.

In some examples, a light feature pattern may indicate a direction ofmovement. For example, a light feature may include a plurality ofdiscrete light elements that form the light feature pattern, and theelements may sequentially light up so that the light portions appear tomove along the light feature (e.g., the lights may appear to “run” or“chase” around a ring to indicate rotational movement). In variousexamples, a light feature may indicate a direction that a part of adevice (e.g., an arm link) should be manually moved, or a light featuremay indicate a direction that the part will move automatically whenactivated, or a light feature may indicate a direction that a part ismoving (e.g., in response to a teleoperated control input), whichprovides supporting visual feedback to a user that the user is rotatingor translating a joint in a correct direction.

In some examples, a light feature may communicate a target displacement,a range of motion limit, or a proximity to a range of motion limit. Forexample, a light feature may include a visually different locationfeature (e.g., a visual “hot spot” or bright spot), and a device mayhave an indicator proximate the light feature. The device may beconfigured so that the target or range of motion limit is reached whenthe location feature aligns with the indicator. The indicator may be aphysical feature (e.g., a mark) on a device, or it may be a locationfeature on a light feature.

Example location features are illustrated in FIGS. 20A-B. FIGS. 20A-20Bshow a light feature 2012, which may be at a joint 2016 on the device2000. The joint 2016 may allow the second arm portion 2014 of the device2000 to rotate with respect to the first portion 2002. The light feature2012 may have a location feature 2020, which may be a visual hot spot orother visually different feature, and which may represent a targetlocation or a range of motion limit. The second arm portion 2014 of thedevice 2000 may include an indicator 2022. When the second arm portion2014 of the device 2000 is rotated upward with respect to the firstportion 2002 (or the first portion 2002 is rotated downward with respectto the second arm portion 2014), the indicator 2022 moves toward thelocation feature 2020 and eventually aligns with the location feature2020 when the range of motion limit is reached, as shown in FIG. 20B.

In another example, an adjacent light feature 2010 may include a secondlocation feature 2024 (e.g., hot spot), and the target displacement maybe reached when the second location feature 2024 aligns with the firstlocation feature 2020. The second location feature 2024 or firstlocation feature 2020 or both may change appearance (e.g., change color)when the second location feature 2024 aligns with the first locationfeature 2020. In some examples, the second location feature 2024 may beconsidered aligned with the first location feature 2020 when the secondlocation feature 2024 is within a specified displacement (e.g., aspecified distance (e.g., 2 mm) or angular rotation (e.g., 2 degrees) orpercentage (e.g., 1 percent of range of motion) of the first locationfeature 2020.

In another example, a light feature 2006 may include a third locationfeature 2026 and a fourth location feature 2028, which may represent atarget displacement location or target displacement range, and alignmentof the third location feature 2026 with the fourth location feature 2028may indicate that a target location has been achieved, as shown in FIG.20B. As the third location feature 2026 and fourth location feature 2028merge or overlap, the third location feature 2026, fourth locationfeature 2028, or both, or an overlapping region, may change appearance,such as by changing color, or brightening, or changing pulse state(e.g., change from steady to pulsing). In another example, the fourthlocation feature 2028 may indicate a desired displacement or range ofdisplacements, which may, for example, correspond to a specifiedlocation (e.g., center) between available range of motion limits. Therange of motion limits may be mechanical limits for the joint, or theymay be based upon other constraints such as proximity to other arms or apatient clearance requirement.

Any of the visual location feature examples described above may provideguidance for a user to move a device or device component, such as an armor arm link (e.g., in system 100, 200, or 900), to a target position orrange. A system may include a plurality of light features that eachinclude one or more location features, so that a user may be guidedthrough movement of multiple joints in the system. In some examples, theorder of a sequence of movements at a plurality of joints may beindicated by an appearance (e.g., pulsing or color change or brightnesschange) of a light feature or location feature portion of a lightfeature. A first light feature pattern associated with a first joint tobe moved is first activated until the first joint is in a properdisplacement, at which time a second light feature pattern associatedwith a second joint to be moved is activated until the second joint isin a proper displacement, and so on. In some examples, responsive to achange in pose or configuration, the light feature patterns may beupdated. For example, the position of a location feature that indicatesa joint target location or range may be change to indicate a new targetlocation or target range. Updating the position of one or more locationfeatures may provide guidance to a user regarding how to move an arm (orother device) to achieve a target position or range and may be dynamicbased on the state of one or more other system components, such asanother arm joint that is moved. Dynamic updating of light featurepatterns allows visual guidance to be iterative, so that if a firstlight feature directs a first joint motion, and a second joint featuredirects a second joint motion, the first light feature pattern may beupdated to direct the user to adjust the first joint again based on thesecond joint's movement.

In another example, a light feature 2030 may include a fifth locationfeature 2032 and a sixth location feature 2034, which may indicate ajoint or arm range of motion limit. The fifth location feature 2032 mayabut the sixth location feature 2034 when the range of motion limit isreached, as shown in FIG. 20B. In some examples, the fifth locationfeature 2032, sixth location feature 2034, or both may change appearancewhen the sixth location feature 2034 approaches the fifth locationfeature 2032 (e.g., change color to yellow) or reaches the fifthlocation feature 2032 (e.g., reaches the end of range of motion, whichmay be indicated for example by a change to red, or flashing, or both).In some examples, an entire light feature pattern (or any portionthereof) may change appearance in addition or in alternative to one orboth of the location features 2032, 2034 changing appearance.

Any of the light feature location techniques shown in FIGS. 20A-20B orexplained above may be used individually or in any combination. WhileFIGS. 20A-B show a rotational example, the techniques for indicatingrange of motion limit or target displacement location may also beapplied to different types of motion, such as angular motion (e.g., thearticulation of an arm joint in system 100 or system 900) ortranslational motion (e.g., when a first part slides relative to asecond part to align a light feature with an indicator on a device or toalign with another light feature, the alignment may indicate a range ofmotion limit, or achievement of a target position).

Light Features on a User Control Unit

A user control unit may include one or more light features, which mayindicate information about the control unit or an associated system,such as a user interaction with the user control unit, or a system state(e.g., lock state).

FIG. 17 is an illustration of a user control unit 1700, which mayinclude a display portion 1720 and a user input portion 1730. The usercontrol unit 1700 may, for example, be used to control a manipulator,such as the manipulator 1800 shown in FIGS. 18 and 19 . The user controlunit 1700 and manipulator 1800 may include some or all of the componentsof the system 1200 illustrated in FIG. 12 and described above. The usercontrol unit 1700 may include one or more light features 1702, 1706,which may change appearance (e.g., change color or brightness or flashor pulse) in response to a system state or user interaction with thecontrol unit 1700. For example, one or more light features 1702, 1706may change appearance responsive to a user touch against a face 1740 ofthe control unit 1700 or other portion (e.g., side 1742, or front 1744)of the control unit 1700, or against a user input 1704, 1706. Anappearance of a light feature 1702, 1706 change may indicate a lockstate or control state of the system, such as whether the manipulator1800 is being controlled by the control unit 1700.

As shown in FIG. 17 , the user control unit 1700 may include a firstuser input 1704, which may, for example, be a scroll wheel. In someexamples, the scroll wheel may control insertion and withdrawal of aninstrument, such as a medical catheter. As shown, a first light feature1702 is at the first user input 1704. The light feature 1702 may includean LED, which may be on a top surface 1740 of the control unit 1700, orthe top surface 1740 may include a plurality of holes and the LED panelmay be beneath the top surface 1740 under the holes (e.g., as describedabove in reference to FIG. 3A). The light feature 1702 may extend aroundthe first user input 1704 (e.g., scroll wheel). For example, the lightfeature 1702 may extend part way around the first user input 1704 or allof the way around the first user input 1704. The light feature 1702 maybe configured as a light ring. In some examples, the light feature 1702may indicate a control state. For example, the light feature 1702 maybecome bright when the first user input 1704 is controlling or ready tocontrol a device such as the manipulator 1800 or an attached instrument.In an example, the user control unit 1700 may detect user contact,presence, or proximity (e.g., of a hand), and the light feature 1702 maychange appearance (e.g., change from dim to bright) when user contact,presence, or proximity is detected.

The user control unit 1700 may also include a user input 1708, which mayfor example be a track ball. In some examples, the user control unit1700 and manipulator 1800 may be configured so that the track ballcontrols directional movement (e.g., left, right, up, or down steering)of the manipulator. For example, rolling the track ball to the left maysteer a distal end of a steerable instrument mounted to the manipulatorto the left, rolling the track ball to the right may steer the distalend of the instrument to the right, rolling the track ball forward maysteer the distal end down, and rolling back may steer the instrumentupward. A second light feature 1706 may be at second user input 1708,and second light feature 1706 may be configured similarly to first lightfeature 1702.

In some examples, one or more light features (e.g., 1702, 1706) maypresent a changed appearance in response to detection of a user input orof user contact, presence, or proximity. For example, one or more otherlight features on the user control unit, on a manipulating systemcontrolled by the user control system unit as described below, or both,may change appearance to match the appearance of light feature 1702 whenlight feature 1702 changes in response to detected user input or usercontact, presence, or proximity. As a more specific example, lightfeatures 1702 and 1706 may be rings that are dimly lit blue to indicatethe surgical system is ready for use, and when a clinical user's handcomes near or touches first user input 1704, light features 1702 and1706 change to a brightly lit blue to indicate the system is now underuser control. In addition, one or more similar light features oncorresponding manipulator 1800 optionally may change from a dimly lit toa brightly lit blue light ring to indicate to a clinician (e.g., asecond clinician remote from the user control system) dear themanipulating system that it is under user control. And, when the firstclinician at the user control unit moves the hand away from a userinput, the light features change from the brightly lit blue to the dimlylit blue.

The light features may indicate other system states and events withvarious light feature patterns as described herein. The user controlsystem 150 may also include other light features and light featurepatterns, including controllable display features, as described herein.

Example Light Feature on a Manipulator System

FIG. 18 is an illustration of another example manipulator 1800 withlight features. An instrument 1850 is shown mounted on the manipulator1800. The instrument 1850 may include a proximal mounting portion 1852,an instrument shaft 1854 extending distally from the mounting portion1852, and an instrument steerable distal working portion 1856 coupled tothe distal end of instrument shaft 1854.

The manipulator 1800 may include a first light feature 1802 at a lowerrotational joint between an arm link 1804 and a forearm link 1806 of themanipulator 1800. The forearm link 1806 may rotate relative to the armlink 1804 at a joint, and the light feature 1802 may be located at thejoint. As shown, the long axes of links 1804, 1806 are coaxial anddefine the axis of rotation of link 1806 with reference to link 1804.First light feature 1802 may function as described herein.

The manipulator 1800 may include a second light feature 1808 at aninterface where the instrument 1850 may be mounted to an instrumentcarriage portion 1810 of the manipulator 1800. The second light feature1808 may, for example, indicate a connectivity state of the interface(e.g., instrument physically mounted, energy or communication betweeninstrument and carriage is functioning, etc.) or a control state of theinstrument 1850 or manipulator 1800. The manipulator 1850 mayadditionally or alternatively include a third light feature 1812 on afixed or telescoping spar 1814 of the manipulator 1800, and the thirdlight feature 1812 may function as described herein. The manipulator1800 may include a fourth light feature 1816 at a top portion 1818 ofthe spar 1814, where an instrument adaptor 1820 may be connected.

As described herein, first, second, third, and fourth light features1802, 1808, 1812, and 1816 may function individually or in anycombination to indicate operating conditions or events at acorresponding location on manipulator 1800, or for manipulator 1800 ingeneral, or for the teleoperated surgical system in general (e.g.,including a corresponding control system unit, and optionally one ormore auxiliary support systems).

FIG. 19 is an illustration of the manipulator 1800 on a movablemanipulator system unit 1900. The manipulator system unit 1900 mayinclude a base 1902, a display system 1904 that may for example beconfigured to output information similar to a display on a correspondinguser control system unit, and the manipulator 1800. The manipulator 1800is shown in FIG. 19 with no instrument mounted, and so a first interfaceportion 1908 and second interface portion 1910 are visible. As describedabove, an instrument may be mounted to the manipulator 1800 at the firstinterface portion 1908 and second interface portion 1910, and a lightfeature 1808 may be at (e.g., extend around) the first interface portion1908, a light feature 1816 may be at the second interface portion 1910,or both. The manipulator system unit 1900 may also include lightfeatures at other locations as described above or elsewhere herein(e.g., as shown in FIG. 9A), such as at the bottom of the base (e.g.,near the floor) or where the display system 1904 or the manipulator arm1906 meet the base.

FIG. 21 is an illustration of a medical device 2100 that has atelescoping column 2110 extending from a base 2108, which may forexample be a base of a manipulating system as shown in FIGS. 2A-2B, FIG.9A, or FIG. 19 . The telescoping column 2110 may have one, two, three,four, or more telescoping parts. As shown as an illustration, thetelescoping column 2110 may have a first part 2102, a second part 2104that extends from the first part 2102, and a third part 2106 thattelescopes from the second part 2104. An arm, spar, or other device maybe mounted to the distal third part 2106. The device 2100 may have aplurality of light features (e.g., light rings), which may correspond tothe first part 2102, second part 2104, and third part 2106. For example,an outer ring 2112 may correspond to the first part 2102, a middle ring2114 may correspond to the second part 2104, and an inner ring 2116 maycorrespond to the third part 2106. As shown, the rings are concentricaround the column 2110, and the correlation between the relative sizeand position of the rings and the telescoping parts of the column 2110may assist a user with understanding the relationship between the ringsand telescoping parts. The rings may communicate information about thetelescoping parts, such as rotational or translational range of motion(e.g., red indicates end of range of motion), target joint orientations,target telescoping link part translations, etc., as described herein.While three rings and corresponding telescoping parts are shown, more(e.g., 4, 5, or more) or fewer (e.g., 2) pairs of rings andcorresponding parts could also be used.

FIG. 22 is an illustration of an example light feature 2220 at a swivelconnector 2210, which may couple to an endotracheal tube that extendsinto a patient 2201. The light feature 2220 may extend along a side 2232of a mount portion of a manipulating system 2200 near the swivelconnector 2210. In some examples, the light feature 2220 may extendaround the mount portion to form a light ring. The mount portion mayhold, for example, a guiding part for a flexible instrument catheter, aninstrument cannula, or as shown an endotracheal tube swivel connector.The light appearance of the light feature 2220 (e.g., color or otherlight feature pattern) may indicate, for example, whether the swivelconnector 2210 is in alignment (e.g., green or steady light) or out ofalignment (e.g., red or pulsing may indicate out of alignment). And soin this example, a light feature is used to indicate the state of aninterface connection. Further, the light feature optionally may be usedto indicate the status of respiratory air flow through the swivelconnector 2210 as an example of a flux through an interface.

FIGS. 23A and 23B are illustrations of a medical device 2300 that has aplurality of light features 2302, 2304, 2306 around a joint that hasmultiple degrees of freedom. The medical device 2300 may have a firstpart 2308, which may for example be a portion of an arm, or a base, ofthe manipulating system 100, 900, or 1800. The first part 2308 may becoupled to a second part 2310 using a joint, such as a spherical joint,that has multiple degrees of freedom. For example, the joint may rotatein two or more orientation axes, as indicated by arrows in the figure.That is, second part 2310 may roll as indicated by arrow 2310 a, pitchas indicated by arrow 2310 b, and yaw as indicated by arrow 2310 c. Thelight features 2302, 2304, 2306 surround the joint, and optionally maybe concentric with each other. In some examples, a first light feature2302 may correspond to roll motion. The first light feature 2302 mayindicate, for example, direction of roll motion (e.g., an orientationdirection a user should move part 2310, or a direction that a joint ismoving, or a direction the joint is about to move). The light feature2302 may include one or more location features 2322, 2324 (e.g., asdescribed above), which may indicate a current orientation, indicated bylocation feature 2322, relative to a range of motion limit, indicated bylocation feature 2324. Responsive to roll (e.g., counter-clockwise) ofthe second part 2310 relative to the first part 2308, the locationfeature 2322 may move (counter-clockwise) toward the range of motionlimit indicated by location feature 2324.

Light feature 2304 may correspond to rotational pitch and/or yaw motion,either alone or in combination with yaw. For example, responsive toyawing the second part 2310 from the orientation shown in FIG. 23A tothe orientation shown in FIG. 23B, the light feature 2304 may changeappearance (e.g., may change from green to yellow to indicate theapproach of a range of motion limit). In some examples, a locationfeature 2314 on the light feature 2304 may change appearance. Forexample, the location feature 2314 may change to yellow, but the rest ofthe light feature 2304 may stay green.

In some examples, the light features 2304 and 2306 may both correspondto rotational motion. For example, the location feature 2314 maycorrespond to a first angular orientation (e.g., a present orientation),and a location feature 2316 on light feature 2306 may indicate thedesirability or proximity to range of motion if the part 2310 is furtherrotated in a direction aligned with light the feature 2306.

While the light features 2302, 2304, 2306 are shown and described asdiscrete light features (e.g., separate rings), they also may bedifferent portions of the same light feature. Further, it can be seenthat one or more similar light feature displays can be adapted for usewith translational motion of second part 2310 with reference to firstpart 2308, showing second part 2310's position in one, two, or threeCartesian DOFs in order to indicate translational information analogousto the rotational information described herein. And, it can be seen thatone or more similar light features can be adapted for use with anycombination of one, two, or three Cartesian orientations and one, two,or three Cartesian positions.

FIG. 24 is a flowchart illustration of a method 2400 of controlling amedical device having an interface. The method 2400 may include, at 2402presenting a first illumination state of the light feature correspondingto a first state of the interface. The method 2400 may include, at 2402,presenting a second illumination state of the light featurecorresponding to a second state of the interface. In some examples, thefirst illumination state corresponds to a first joint displacement valueand the second illumination state corresponds to a second jointdisplacement value. In some examples, the first illumination statecorresponds to a first joint angle and the second illumination statecorresponds to a second joint angle. In some examples, presenting thefirst illumination state includes presenting a first animated state inwhich an apparent movement of the light feature corresponds to a firstmovement at the interface, and presenting a second illumination stateincludes presenting a second animated state in which an apparentmovement of the light feature corresponds to a second movement at theinterface.

FIG. 25 is a flowchart illustration of a method 2500 of controlling anintegrated display on a teleoperated surgical system. The method 2500may include, at 2502, activating a display on a structural component ofthe teleoperated surgical system to present information on thestructural component. Activating the display may include, for example,activating a lighted display behind an array of micro-holes in thestructural component.

The method 2500 may include, at 2504, receiving information related to asurgical procedure. Receiving information may include receivinginformation from a clinician's console and presenting informationincludes presenting a message or instruction on the integrated display.

The method 2500 may include, at 2506, presenting information on theintegrated display based on the received information. In an example, theintegrated display is on a first manipulator arm and presentinginformation includes presenting a first identifier for the firstmanipulator arm. The method may also include presenting a secondidentifier on a second manipulator arm, wherein the first manipulatorarm is differentiable from the second manipulator arm based on thepresentation of first identifier and the second identifier.

In some examples, the received information may include, for example, apresent pose of a manipulator arm and the presented information includesan operating mode or status associated with the present pose.

In some examples receiving information may include receiving an elapsedtime or a time to completion, and presenting information on theintegrated display may include presenting the elapsed time or the timeto completion.

In some examples, receiving information may include receiving aninstrument type that is coupled to a manipulator arm and presentinginformation includes presenting the instrument type.

In various examples, the method may include indicating a safe grab pointon an arm, indicating to interact with to manipulate an arm, presentingguided setup instructions on the integrated display, identifying an armto service or adjust, indicating an instrument change to be performed onan identified manipulator arm, or communicating a range of motionstatus.

The method 2500 may include, at 2508, deactivating the display when thedisplay is not in use, wherein the display is not visually detectablewhen the display is deactivated.

FIG. 26 is a flowchart illustration of a method 2600 of interacting witha user via a user interface on a teleoperated surgical system. Themethod 2600 may include, at 2602, presenting information on a first userinterface coupled to an arm of the teleoperated surgical system. Themethod 2600 may include, at 2604, detecting a change in orientation ofthe arm. Detecting a change in orientation of the arm may include, forexample, receiving information from an orientation sensor. The method2600 may include, at 2606, adjusting an orientation of the informationpresented on the user interface based on the detected change inorientation of the arm.

The method 2600 may include, at 2608, presenting an indication on theuser interface of a location of second input device. The method may alsoinclude receiving user input through the second input device.

The method 2600 may include, at 2610, locking or unlocking a jointresponsive to receiving user input (e.g., responsive to receiving userinput through the second input device.)

FIG. 27 is a flowchart illustration of a method 2700 of controlling aconfiguration of a manipulator arm and a handle of a teleoperatedsurgical system. The method may include, at 2702 moving a manipulatorarm to a first pose.

The method 2700 may include, at 2704 automatically adjusting anorientation of a handle to a specified orientation responsive tomovement of the manipulator arm to the first pose. Adjusting theorientation of the handle includes positioning the handle in aconsistent orientation through a range of positions of the manipulatorarm. The method may include, for example, positioning the handle in ahorizontal orientation.

In some examples, the method may include determining a handleorientation based on a frame of reference. The frame of reference mayinclude one or more user parameters. For example, the one or more userparameters includes a user height, and the method may includepositioning the handle at an orientation based at least in part on theuser height.

The method 2700 may include, at 2706, receiving a user input. In someexamples, a specified orientation of the handle may be determined basedat least in part on the user input.

In some examples, the method 2700 may include, at 2708, performing anaction responsive to the input. For example, the method 2700 may includeperforming an egress procedure responsive to a user input and adjustingthe orientation of the handle to an egress orientation as part of theegress procedure. In some examples, the method 2700 may includereleasing the handle responsive to the user input, wherein theorientation of the handle is manually adjustable when the handle isreleased. In some examples, the method may include receiving a firstuser input indicating a first handle orientation when the arm is a firstpose, and receiving a second user input indicating a second handleorientation when the arm is in a second pose, wherein a specified thirdorientation of the handle for a third pose of the arm is determinedbased at least in part on the first user input and the second userinput.

Although aspects are in some instances described in the context of amodular, single-arm units of a manipulating system as part of acomputer-assisted teleoperated surgical system, in other embodimentsthese aspects may be incorporated into one or more arms that are mountedon a surgical table (table base or side-rail), or on another object inthe operating room, or are mounted on the floor or the ceiling or a wallof the operating room. In some examples, two or more arms with theseaspects may be mounted on a manipulating system, or on another object inthe operating room, or at a single location on a table, object, floor,ceiling, or wall in the operating room.

In addition, two single-arm manipulating systems located together aredisclosed as examples, but disclosed aspects apply to more than twosingle-arm manipulating systems, such as three, four, or five single-armmanipulating systems, or to a combination of any number of single-armmanipulating systems and one or more multi-arm manipulating systems,such as a da Vinci® surgical system, commercialized by IntuitiveSurgical, Sunnyvale, Calif.

Persons of skill in the art will understand that any of the featuresdescribed above may be combined with any of the other example features,as long as the features are not mutually exclusive. All possiblecombinations of features are contemplated, depending on clinical orother design requirements. In addition, if manipulating system units arecombined into a single system (e.g., telesurgery system), eachindividual unit may have the same configuration of features, or, onemanipulating system may have one configuration of features and anothermanipulating system may have a second, different configuration offeatures. As stated above, an exhaustive list of all possiblecombinations of aspects and features would result in a prolixdescription, and it should be clear to skilled persons that variouscombinations of the aspects and features described herein arecontemplated.

The examples (e.g., methods, systems, or devices) described herein maybe applicable to surgical procedures, non-surgical medical procedures,diagnostic procedures, cosmetic procedures, and non-medical proceduresor applications. The examples may also be applicable for training or forobtaining information, such as imaging procedures. The examples may beapplicable to handling of tissue that has been removed from human oranimal anatomies and will not be returned to a human or animal, or foruse with human or animal cadavers. The examples may be used forindustrial applications, general robotic uses, manipulation ofnon-tissue work pieces, as part of an artificial intelligence system, orin a transportation system.

The above detailed description includes references to the accompanyingdrawings, which form a part of the detailed description. The drawingsshow, by way of illustration, specific embodiments in which theinvention may be practiced. These embodiments are also referred toherein as “examples.” Such examples may include elements in addition tothose shown or described. However, the present inventors alsocontemplate examples in which only those elements shown or described areprovided. Moreover, the present inventors also contemplate examplesusing any combination or permutation of those elements shown ordescribed (or one or more aspects thereof), either with respect to aparticular example (or one or more aspects thereof), or with respect toother examples (or one or more aspects thereof) shown or describedherein.

In the event of inconsistent usages between this document and anydocuments so incorporated by reference, the usage in this documentcontrols.

In this document, the terms “a” or “an” are used, as is common in patentdocuments, to include one or more than one, independent of any otherinstances or usages of “at least one” or “one or more.” In thisdocument, the term “or” is used to refer to a nonexclusive or, such that“A or B” includes “A but not B,” “B but not A,” and “A and B” unlessotherwise indicated. In this document, the terms “including” and “inwhich” are used as the plain-English equivalents of the respective terms“comprising” and “wherein.” Also, in the following claims, the terms“including” and “comprising” are open-ended, that is, a system, device,article, composition, formulation, or process that includes elements inaddition to those listed after such a term in a claim are still deemedto fall within the scope of that claim. Moreover, in the followingclaims, the terms “first,” “second,” and “third,” etc. are used merelyas labels, and are not intended to impose numerical requirements ontheir objects.

Geometric terms, such as “parallel”, “perpendicular”, “round”, or“square”, are not intended to require absolute mathematical precision,unless the context indicates otherwise. Instead, such geometric termsallow for variations due to manufacturing or equivalent functions. Forexample, if an element is described as “round” or “generally round”, acomponent that is not precisely circular (e.g., one that is slightlyoblong or is a many-sided polygon) is still encompassed by thisdescription. Coordinate systems or reference frames are provided foraiding explanation, and implantations may use other reference frames orcoordinate systems other than those described herein.

The above description is intended to be illustrative, and notrestrictive. For example, the above-described examples (or one or moreaspects thereof) may be used in combination with each other. Otherembodiments may be used, such as by one of ordinary skill in the artupon reviewing the above description. The Abstract is provided to allowthe reader to quickly ascertain the nature of the technical disclosure.It is submitted with the understanding that it will not be used tointerpret or limit the scope or meaning of the claims. Also, in theabove Detailed Description, various features may be grouped together tostreamline the disclosure. This should not be interpreted as intendingthat an unclaimed disclosed feature is essential to any claim. Rather,inventive subject matter may lie in less than all features of aparticular disclosed embodiment. Thus, the following claims are herebyincorporated into the Detailed Description as examples or embodiments,with each claim standing on its own as a separate embodiment, and it iscontemplated that such embodiments may be combined with each other invarious combinations or permutations. The scope of the invention shouldbe determined with reference to the appended claims, along with the fullscope of equivalents to which such claims are entitled.

What is claimed is:
 1. A medical device comprising: a first componentcomprising an interface; a light feature surrounding at least part ofthe interface; and a controller coupled to the light feature andcomprising a memory in which are stored instructions for: the controllercausing a first illumination state of the light feature corresponding toa first state of the interface, and the controller causing a secondillumination state of the light feature corresponding to a second stateof the interface, wherein the first illumination state and the secondillumination state are indicative of at least one of a movement of or aspatial relationship between two links connected at the interface. 2.The medical device of claim 1, wherein the first state includes a firstdisplacement value and the second state includes a second displacementvalue, and an aspect of the first illumination state of the lightfeature corresponds to the first displacement value, and an aspect ofthe second illumination state of the light feature corresponds to thesecond displacement value.
 3. The medical device of claim 2, wherein thefirst illumination state comprises a first illuminated length of thelight feature corresponding to the first displacement value at theinterface, and the second illumination state comprises a secondilluminated length of the light feature corresponding to a second jointdisplacement value at the interface.
 4. The medical device of claim 1,wherein an aspect of the first illumination state of the light featurecorresponds to a first rotational state of the interface, and an aspectof the second illumination state corresponds to a second rotationalstate or a first translational state of the interface.
 5. The medicaldevice of claim 4, wherein the light feature indicates position relativeto a range of motion limit.
 6. The medical device of claim 1, wherein afirst aspect of the light feature corresponds to a rotational state anda second aspect of the light feature corresponds to a translationalstate.
 7. The medical device of claim 6, wherein a length of the lightfeature in a transverse direction corresponds a translationaldisplacement value, and a length of the light feature extending aroundthe interface corresponds to a rotational displacement value.
 8. Themedical device of claim 1, wherein the light feature includes movablefeatures and a first rate of movement of the movable features in thefirst illumination state corresponds to a first velocity of movement ofa movable portion of the medical device, and a second rate of movementof the movable features in the second illumination state corresponds toa second velocity of movement of the movable portion of the medicaldevice.
 9. The medical device of claim 1, wherein the light featureindicates a kinematic pose of the medical device.
 10. The medical deviceof claim 1, wherein the light feature indicates a source of controlcommands for the medical device.
 11. A teleoperated surgical systemcomprising: a first link; a second link coupled to the first link at aninterface; a light feature surrounding at least part of the interfaceand having a variable illumination state; and a controller coupled tothe light feature and comprising a memory in which are storedinstructions for controlling the illumination state of the light featurebased at least in part on a movement or a spatial relationship of thesecond link with respect to the first link.
 12. The teleoperatedsurgical system of claim 11, wherein the controller includes storedinstructions to control the illumination state of the light featurebased on an angular relationship of the second link with respect to thefirst link.
 13. The teleoperated surgical system of claim 11, whereinthe controller includes stored instructions to control the illuminationstate of the light feature based on a kinematic pose of the first linkand the second link.
 14. The teleoperated surgical system of claim 11,wherein the controller includes stored instructions to control theillumination state of the light feature based on a range of motion. 15.The teleoperated surgical system of claim 11, wherein controllerincludes instructions for: the controller causing a first illuminationstate of the light feature corresponding to a first state of theinterface, and the controller causing a second illumination state of thelight feature corresponding to a second state of the interface.
 16. Theteleoperated surgical system of claim 15, wherein the first stateincludes a first displacement value and the second state includes asecond displacement value, and an aspect of the first illumination stateof the light feature corresponds to the first displacement value, and anaspect of the second illumination state of the light feature correspondsto the second displacement value.
 17. The medical device of claim 4,wherein the light feature includes movable features and a first rate ofmovement of the movable features in the first illumination statecorresponds to a first velocity of movement of a movable portion of themedical device, and a second rate of movement of the movable features inthe second illumination state corresponds to a second velocity ofmovement of the movable portion of the medical device.
 18. The medicaldevice of claim 4, wherein the light feature indicates a kinematic poseof the medical device.
 19. The medical device of claim 4, wherein thelight feature indicates a source of control commands for the medicaldevice.
 20. The medical device of claim 6, wherein the light featureincludes movable features and a first rate of movement of the movablefeatures in the first illumination state corresponds to a first velocityof movement of a movable portion of the medical device, and a secondrate of movement of the movable features in the second illuminationstate corresponds to a second velocity of movement of the movableportion of the medical device.
 21. The medical device of claim 6,wherein the light feature indicates a kinematic pose of the medicaldevice.
 22. The medical device of claim 6, wherein the light featureindicates a source of control commands for the medical device.